Review: Sorceress of Darshiva, by David Eddings

Series:	The Malloreon #4
Publisher:	Del Rey
Copyright:	December 1989
Printing:	November 1990
ISBN:	0-345-36935-1
Format:	Mass market
Pages:	371

This is the fourth book of the Malloreon, the sequel series to the Belgariad. Eddings as usual helpfully summarizes the plot of previous books (the one thing about his writing that I wish more authors would copy), this time by having various important people around the world briefed on current events. That said, you don't want to start reading here (although you might wish you could). This is such a weird book. One could argue that not much happens in the Belgariad other than map exploration and collecting a party, but the party collection involves meddling in local problems to extract each new party member. It's a bit of a random sequence of events, but things clearly happen. The Malloreon starts off with a similar structure, including an explicit task to create a new party of adventurers to save the world, but most of the party is already gathered at the start of the series since they carry over from the previous series. There is a ton of map exploration, but it's within the territory of the bad guys from the previous series. Rather than local meddling and acquiring new characters, the story is therefore chasing Zandramas (the big bad of the series) and books of prophecy. This could still be an effective plot trigger but for another decision of Eddings that becomes obvious in Demon Lord of Karanda (the third book): the second continent of this world, unlike the Kingdoms of Hats world-building of the Belgariad, is mostly uniform. There are large cities, tons of commercial activity, and a fairly effective and well-run empire, with only a little local variation. In some ways it's a welcome break from Eddings's previous characterization by stereotype, but there isn't much in the way of local happenings for the characters to get entangled in. Even more oddly, this continental empire, which the previous series set up as the mysterious and evil adversaries of the west akin to Sauron's domain in Lord of the Rings, is not mysterious to some of the party at all. Silk, the Drasnian spy who is a major character in both series, has apparently been running a vast trading empire in Mallorea. Not only has he been there before, he has houses and factors and local employees in every major city and keeps being distracted from the plot by his cutthroat capitalist business shenanigans. It's as if the characters ventured into the heart of the evil empire and found themselves in the entirely normal city next door, complete with restaurant recommendations from one of their traveling companions. I think this is an intentional subversion of the normal fantasy plot by Eddings, and I kind of like it. We have met the evil empire, and they're more normal than most of the good guys, and both unaware and entirely uninterested in being the evil empire. But in terms of dramatic plot structure, it is such an odd choice. Combined with the heroes being so absurdly powerful that they have no reason to take most dangers seriously (and indeed do not), it makes this book remarkably anticlimactic and weirdly lacking in drama. And yet I kind of enjoyed reading it? It's oddly quiet and comfortable reading. Nothing bad happens, nor seems very likely to happen. The primary plot tension is Belgarath trying to figure out the plot of the series by tracking down prophecies in which the plot is written down with all of the dramatic tension of an irritated rare book collector. In the middle of the plot, the characters take a detour to investigate an alchemist who is apparently immortal, featuring a university on Melcena that could have come straight out of a Discworld novel, because investigating people who spontaneously discover magic is of arguably equal importance to saving the world. Given how much the plot is both on rails and clearly willing to wait for the protagonists to catch up, it's hard to argue with them. It felt like a side quest in a video game. I continue to find the way that Eddings uses prophecy in this series to be highly amusing, although there aren't nearly enough moments of the prophecy giving Garion stage direction. The basic concept of two competing prophecies that are active characters in the world attempting to create their own sequence of events is one that would support a better series than this one. It's a shame that Zandramas, the main villain, is rather uninteresting despite being female in a highly sexist society, highly competent, a different type of shapeshifter (I won't say more to avoid spoilers for earlier books), and the anchor of the other prophecy. It's good material, but Eddings uses it very poorly, on top of making the weird decision to have her talk like an extra in a Shakespeare play. This book was astonishingly pointless. I think the only significant plot advancement besides map movement is picking up a new party member (who was rather predictable), and the plot is so completely on rails that the characters are commenting about the brand of railroad ties that Eddings used. Ce'Nedra continues to be spectacularly irritating. It's not, by any stretch of the imagination, a good book, and yet for some reason I enjoyed it more than the other books of the series so far. Chalk one up for brain candy when one is in the right mood, I guess. Followed by The Seeress of Kell, the epic (?) conclusion. Rating: 6 out of 10

A look at the 2021 model of Dell XPS 13 - available with Linux pre-installed

I received a new laptop for work - a Dell XPS 13. Dell has been long famous for offering certain models with pre-installed Linux as a supported option, and opting for those is nice for moving some euros/dollars from certain PC desktop OS monopoly towards Linux desktop engineering costs. Notably Lenovo also offers Ubuntu and Fedora options on many models these days (like Carbon X1 and P15 Gen 2).

Obviously a smooth, ready-to-rock Ubuntu installation is nice for most people already, but I need openSUSE, so after checking everything is fine with Ubuntu, I continued to install openSUSE Tumbleweed as a dual boot option. As I m a funny little tinkerer, I obviously went with some special things. I wanted:

• Ubuntu to remain as the reference supported OS on a small(ish) partition, useful to compare to if trying out new development versions of software on openSUSE and finding oddities.
• openSUSE as the OS consuming most of the space.
• LUKS encryption for openSUSE without LVM.
• ext4 s new fancy fast_commit feature in use during filesystem creation.
• As a result of all that, I ended up juggling back and forth installation screens a couple of times (even more than shown below, and also because I forgot I wanted to use encryption the first time around).

First boots to pre-installed Ubuntu and installation of openSUSE Tumbleweed as the dual-boot option:

(if the embedded video is not shown, use a direct link)

Some notes from the openSUSE installation:

• openSUSE installer s partition editor apparently does not support resizing or automatically installing side-by-side another Linux distribution, so I did part of the setup completely on my own.
• Installation package download hanged a couple of times, only passed when I entered a mirror manually. On my TW I ve also noticed download problems recently, there might be a problem with some mirror I need to escalate.
• The installer doesn t very clearly show encryption status of the target installation - it took me a couple of attempts before I even noticed the small encrypted column and icon (well, very small, see below), which also did not spell out the device mapper name but only the main partition name. In the end it was going to do the right thing right away and use my pre-created encrypted target partition as I wanted, but it could be a better UX. Then again I was doing my very own tweaks anyway.
• Let s not go to the details why I m so old-fashioned and use ext4 :)
• openSUSE s installer does not work fine with HiDPI screen. Funnily the tty consoles seem to be fine and with a big font.
• At the end of the video I install the two GNOME extensions I can t live without, Dash to Dock and Sound Input & Output Device Chooser.

I often get intimidated by the brilliance of others by just looking at them from a distance seeing how radiantly they gleam in their knowledge. I often feel like I'm not doing enough or I'm just meant to be a mediocre and there's nothing I can do to be more than that. I often feel like everyone except me is a genius, and are impeccable unlike me. But maybe I'm just at a point these people had previously been at, and have now gone past. Maybe it's my time to steer the wheel they once steered, or maybe they aren't even what they seem to me to be and it's all just a misrepresentation cooked up by an imperfect mind. Hmm...it has to be that. A perfect portrait painted by an imperfect mind which when found out renders every stroke in the portrait imperfect. It has to be that or all. No one starts having it all figured out. A king is born in a day, but a king is not nutured in a day. We all have something we struggle with, it's just different all across. And because their current struggle isn't my current struggle and mine not theirs doesn't make them immuned to the system. Most times all that gleaming in thier knowledge is a collective thing and you only notice when you take a closer look; that there are more heads to the knowledge and brilliance than you think. We mostly see what people know and seldom see what they don't know. If it was the other way around, we'll see how ridiculous the things these highly esteemed people don't know are. What matters most is that they keep making it through their struggles. No man is an island! The sooner we realize this the better. We can always take those things we struggle with to others and work collectively on them, learn from a lot other experiences and gleam collectively. It's like a tug of war. When what is on the other end is more than we can pull off, we invite many more hands to pull it off.

Review: Raybearer, by Jordan Ifueko

Series:	Raybearer #1
Publisher:	Amulet Books
Copyright:	2020
ISBN:	1-68335-719-1
Format:	Kindle
Pages:	308

Tarisai was raised alone in Bhekina House by an array of servants and tutors who were not allowed to touch her. Glimpses of the world were fleeting and sometimes ended by nailed-shut windows. Her life revolved around her rarely-seen mother, The Lady, who treats her with deep affection but rarely offers a word of praise, instead only pushing her to study harder. The servants whispered behind her back (but still in her hearing) that she was not human. At the age of seven, in a child's attempt to locate her absent mother by sneaking out of the house, she finds her father and is told a piece of the truth: she is the daughter of the Lady and a captive ehru, a djinn. At the age of eleven she's sent with two guardians to Oluwan City, the capital of Aritsar, to enter a competition she knows nothing about, for reasons no one has ever explained. Raybearer is a young adult fantasy novel, the first of a duology. Like a lot of young adult novels, it is a coming of age story that follows Tarisai from the end of her highly manipulated childhood through her introduction to a world she was carefully never taught about. Like a lot of young adult fantasy novels, Tarisai has some unusual abilities. What those are, and why she has them, is perhaps less obvious than it may appear at first. Unlike a lot of young adult fantasy novels, Raybearer is not set in a facsimile of Western Europe, the structure of gods and religion is not obviously derived from Christianity or Greek or Norse mythology, and neither Tarisai nor most of the characters of this story are white. Some of the characters are; Ifueko draws from a grab bag of cultures that does include European as well as African, Middle Eastern, and Asian. But the food, the physical descriptions, the landscape, and the hair and hair styles feel primarily African not in the sense of specific identifiable regions, but in the same way that most fantasy feels European even if the map isn't recognizable. That gives this story a freshness that I found delightful. The mythology of this world shares some similarities to standard fantasy tropes, including a bargain with the underworld that plays a similar role to fae bargains in some European fantasy, but it also goes in different directions and finds atypical balances, which gave the story room to catch me by surprise. The magical center of this book (and series), which Tarisai is carefully not told about until the story starts, is a system for anointing and protecting the emperor: selection of people who swear loyalty to him and each other and become his innermost circle, and thereby grant him magical protection. The emperor himself is the Raybearer, possessing an artifact that makes him invulnerable to one form of death for each member of his council he anoints. At eleven council members, he becomes invulnerable to anything but old age, or an attack from one of the council themselves. As the reader learns early in the book, that last part is important. Tarisai is an assassin; her mother's goal is for her to be selected as a member of the council for the prince, who will become the next emperor. But there is rather more to this system of magic than it may first appear, in a way that adds good depth to the mythology. And there is quite a bit more to Tarisai herself than anyone expects. Tarisai as a protagonist follows a more typical young adult pattern, but it's a formula that works for me. Her upbringing isolated from any other children has left her craving connection, but it also made her self-reliant, stubborn, and good at keeping her own counsel. One of the things that I loved about this book is that she's not thrown into a nest of vipers and cynical politics. Some of that is happening in the background, but the first step of her mother's plan is for her to earn the trust of the prince in a competition with other potential council members, all of whom are, well, kids. They fight (some), but they also make friends, helped along by the goal and requirement that they join a cooperative council or be sent home. That gives the plot a more collaborative and social feel than one would otherwise expect from the setup. Ifueko does a great job juggling a challenging cast size by focusing on a few kids with whom Tarisai strikes up a friendship but giving the others distinct-enough personalities that their presence is still felt in the story. There are two character dynamics that stand out: Tarisai's relationship with Prince Ekundayo, and her friendship with Sanjeet. The first carries much of the weight of the plot, of course; Tarisai is supposed to gain his trust and then kill him, and the reader will be unsurprised that this takes twists and turns no one expected. But Ifueko, refreshingly, does not reach for the stock plot development of a romance to complicate matters, even though many of the characters expect that. To the contrary, this is a rare story that at least hints at an acknowledgment that some people are not interested in romance at all, and there are other forms that mutual respect can take. Tarisai's relationship with Sanjeet is a different type of depth: two kids with very different histories finding a common understanding in the ways that they were both abused, and create space for each other. It's a great friendship that includes some deeply touching moments. It took me a bit to get into this book, but once Tarisai starts finding her feet and navigating her new relationships, I was engrossed. The story takes a sharp and nasty turn that was hard to read, but Ifueko chooses to turn it into a story of resiliency rather than survival, which makes it much easier to read than it could have been. She also pulls off the kind of plot that complicates and deepens the motives of the obvious villains in a way that gives the story much greater heft, but without disregarding the damage that they have done. I think the plot did fall apart a bit at the end of the book, with too much quick travel and world-building revelations at the cost of development of the relationships that were otherwise at the center of the book, but I'm hoping the sequel will pull those threads back together. And it's so refreshing to read a fantasy novel of this type with a different setting. It's not perfect: Ifueko falls back on Planet of the Hats regional characterization in a few places, and Songland is so obviously Korea that it felt jarring and out of place. Christianity also snuck its nose into the world-building tent near the end in ways that bugged me a bit, although it was subtle enough that I think most readers won't notice. But compared to most fantasy settings, it feels original and fresh. More of this! Ifueko starts this book with a wonderfully memorable dedication:

For the kid scanning fairy tales for a hero with a face like theirs. And for the girls whose stories we compressed into pities and wonders, triumphs and cautions, without asking, even once, for their names.

I think she was successful on both parts of that promise, and it makes for some great reading. Recommended. Followed by Redemptor. Rating: 8 out of 10

Review: The Last Graduate, by Naomi Novik

Series:	The Scholomance #2
Publisher:	Del Rey
Copyright:	2021
ISBN:	0-593-12887-7
Format:	Kindle
Pages:	388

This is a direct sequel to A Deadly Education, by which I mean it starts in the same minute at which A Deadly Education ends (and let me say how grateful I am for a sequel that doesn't drop days, months, or years between books). You do not want to read this series out of order. This book is also very difficult to review without spoiling either it or the previous book, so please bear with me if I'm elliptical in my ravings. Because The Last Graduate is so good. So good, not only as a piece of writing, but as a combination of two of my favorite tropes in fiction, one of which I can't talk about because of spoilers. I adored this book in a way that is not entirely rational. I will attempt a review below anyway, but if you liked the first book, just stop reading here and go read the second one. It's more of everything I loved in the first book except even better, it did some things I was expecting and some things I didn't expect at all, and it's just so ridiculously good. Just be aware that it has another final-line cliffhanger. The third book is coming in (hopefully) 2022. Novik handles the cliffhanger at the end of the previous book beautifully, which is worth noting because there were so many ways in which it could have gone poorly. One of the best things about this series is Novik's skill at writing El's relationship with her mother, even though her mother has not appeared in the series so far. El argues with her mother's voice in her head, tells stories about her, wonders what her mother would think of her classmates (or in some cases knows exactly what her mother would think of her classmates), and sometimes makes the explicit decision to not be her mother. The relationship has the sort of messy complexity, shared history, and underlying respect that many people experience in life but that I've rarely seen portrayed this well in a fantasy novel. Novik's presentation of that relationship works because El's voice is so strong. Within fifteen minutes of starting The Last Graduate, I was already muttering "I love this book" to myself, mostly because of how much I enjoy El's sarcastic, self-deprecating internal commentary. Novik strikes a balance between self-awareness, snark, humor, and real character growth that rivals Murderbot in its effectiveness of first-person perspective. It carries the story over a few weak points, such as a romance that didn't do much for me. Even when I didn't care about part of the plot, I cared about El's opinion of the plot and what it said about El's growing understanding of how to navigate the world. A Deadly Education was scene and character establishment. El insisted on being herself and following her own morals and social rules, and through that found some allies. The Last Graduate gives El enough breathing space to make more nuanced decisions. This is the part of growing up where one realizes the limitations of one's knee-jerk reactions and innate moral judgment. It's also when it becomes hard to trust success that is entirely outside of one's previous experience. El was not a kid who had friends, so she doesn't know what to do with them now that she has them. She's barely able to convince herself that they are friends. This is one of the two fictional tropes I mentioned, the one that I can talk about (at least briefly) without major spoilers. I have such a soft spot for stubborn, sarcastic, principled characters who refuse to play by the social rules that they think are required to make friends and who then find friends who like them for themselves. The moment when they start realizing this has happened and have no idea how to deal with it or how to be a person who has friends is one I will happily read over and over again. I enjoyed this book from the beginning, but there were two points when it grabbed my heart and I was all in. The first one is a huge spoiler that I can't talk about. The second was this paragraph:

[She] came round to me and put her arm around my waist and said under her breath, "Hey, she can be taught," with a tease in her voice that wobbled a little, and when I looked at her, her eyes were bright and wet, and I put my arm around her shoulders and hugged her.

You'll know it when you get there. The Last Graduate also gives the characters other than El and Orion more room, which is part of how it handles the chosen one trope. It's been obvious since early in the first book that Orion is a sort of chosen one, and it becomes obvious to the reader that El may be as well. But Novik doesn't let the plot focus only on them; instead, she uses that trope to look at how alliances and collective action happen, and how no one can carry the weight by themselves. As El learns more and gains power, she also becomes less central to the plot resolution and has to learn how to be less self-reliant. This is not a book where one character is trained to save the world. It's a book where she manages to enlist the support of a kick-ass project manager and becomes part of a team. Middle books of a trilogy are notoriously challenging. Often they're travel books: the first book sets up a problem, the second book moves the characters both physically and emotionally into a position to solve the problem, and the third book is the payoff. Travel books often sag. They can feel obligatory but somewhat boring, like a chore on the way to the third-book climax. The Last Graduate is not a travel book; it is, instead, a pivot book, which is my favorite form of trilogy. It's a book that rewrites the problem the first book set up, both resolving it and expanding the scope beyond what the reader had expected. This is immensely satisfying when done well, and Novik does it extremely well. This is not a flawless book. There are some pacing hiccups, there is a romance angle that didn't work for me (although it does arrive at some character insights that I thought were spot on), and although I think Novik is doing something interesting with the trope, there is a lot of chosen one power escalation happening here. It's not the sort of book that I can claim is perfectly written. Instead, it's the sort of book that uses some of my favorite plot elements and emotional beats in such an effective way and with such a memorable character that I do not have it in me to care about any of the flaws. Your mileage may therefore vary, but I would be happy to read books like this until the end of time. As mentioned above, The Last Graduate ends on another cliffhanger. This time I was worried that Novik might have ended the series there, since there's enough of an internal climax that I could imagine some literary fiction (which often seems allergic to endings) would have stopped here. Thankfully, Novik's web site says this is not the case. The next year is going to be a difficult wait. The third book of this series is going to be incredibly difficult to write, and I hope Novik is up to the challenge she's made for herself. But she handled the transition between the first and second book so well, and this book is so good that I have a lot of hope. If the third book is half as good as I'm hoping, this is going to be one of my favorite fantasy series of all time. Followed by an as-yet-untitled third book. Rating: 10 out of 10

For some months now I ve been working on some patches to Consfigurator to add support for Linux containers. My goal is to make Consfigurator capable of both performing the initial setup of a container and of entering the running container to apply configuration. For the case of unprivileged LXCs running as non-root, my work-in-progress branch can now do both of these things. As Consfigurator enters the container directly using system calls, it should be decently fast at configuring multiple containers on a host, and it will also be possible to have it do this in parallel. The initial setup for the container uses Consfigurator s existing support for building root filesystems, and it should be easy to extend that to support arbitrary GNU/Linux distributions by teaching Consfigurator how to invoke bootstrapping tools other than debootstrap(8). Here s an example:

(defhost lxc1.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (basic-props)
  (apt:installed "systemd" "netcat")
  (apache:https-vhost ...))
(defhost lxctest.laptop.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (apt:proxy "http://192.168.122.1:3142")
  (basic-props)
  (apt:installed "linux-image-amd64" "lxc")
  (lxc:usernet-usable-by "spwhitton" "lxcbr0")
  (lxc:user-containers-autostart "spwhitton")
  (lxc:user-container-for '(:additional-lines
                            ("lxc.net.0.type = veth"
                             "lxc.net.0.flags = up"
                             "lxc.net.0.link = lxcbr0"
                             ...))
                          "spwhitton"
                          lxc1.silentflame.com))
(defhost laptop.silentflame.com ()
  ...
  (libvirt:kvm-boots-chroot-for '(:always-deploys t)
                                lxctest.laptop.silentflame.com))

This code is a simplified definition of my testing setup for this work. It defines three hosts: a container lxc1, a container host lxctest, and my laptop. When Consfigurator is asked to deploy the laptop, it will set up the root filesystem for lxctest and then boot it as a KVM virtual machine. Preparing that root filesystem will include setting up the root filesystem for lxc1, too, including shifting the ownership and ACLs to match the user namespace LXC will use when booting the container. Thus, once the deployment of the laptop is finished, it will be possible to boot the lxctest VM, connect to it as the user spwhitton, and start lxc1. Consfigurator includes only minimal support for setting up container networking, as there are so many different ways in which you might want to do it. In my own consfig I ve been developing properties to connect containers directly to my tinc VPN. A single tinc daemon runs on the container host, and other tinc daemons route a whole subnet, containing the addresses for each of the containers, to the container host s tinc daemon. As the LXCs Consfigurator sets up run as non-root, some sort of setuid facility is required to configure this networking. Consfigurator s ability to dump executable Lisp images is helping here. I define a function which runs as root to set up the networking:

(defun route-athenet-container-veth (host)
  (let ((user (getenv "USERV_USER"))
        (peer (getenv "USERV_U_PEER"))
        (ip (car (uiop:command-line-arguments))))
    (unless (string-prefix-p (format nil "veth~D_" (getenv "USERV_UID")) peer)
      (error "~A does not belong to requester." peer))
    (unless (member (cons user ip) (get-hostattrs 'veth-ips host) :test #'equal)
      (error "~A does not have permission to route ~A." user ip))
    (flet ((r (&rest args)
             ;; Explicitly passing nil means UIOP will not invoke a shell.
             (run-program args :force-shell nil)))
      (eswitch ((getenv "USERV_U_HOOK_TYPE") :test #'string=)
        ("up"
         (apply #'r
                "sysctl" "-w"
                "net.ipv6.conf.all.forwarding=1"
                ...)
         (r "ip" "addr" "flush" "dev" peer "scope" "link")
         (r "ip" "-6" "addr" "add" "fe80::1/64" "dev" peer)
         (r "ip" "-6" "route" "add" (strcat ip "/128") "dev" peer)
         ...)
        ("down"
         ...
         (r "ip" "-6" "route" "del" (strcat ip "/128") "dev" peer))))))

and then apply the following property to lxctest to dump an image which will call this function and then exit:

(image-dumped
 "/usr/lib/userv/route-athenet-container-veth"
  (route-athenet-container-veth ,(intern (string-upcase (get-hostname)))))

I m using GNU userv to enable ordinary users to run this image as root, so there there s a small script which converts LXC s LXC_HOOK_* environment variables into appropriate command line arguments to userv(1) such that the function above is able to access that information from its environment (the USERV_U_* variables above). You could just as easily do this with sudo, by giving permission for the relevant LXC_HOOK_* environment variables to survive the switch to root. What s particularly nice about this is that there s no need to write any code to keep a config file updated, specifying which users are allowed to route which IPs to their containers. ROUTE-ATHENET-CONTAINER-VETH receives a HOST value for the container host and can just look at the metadata established by properties for particular containers. Each time this metadata is updated and lxctest is deployed, a fresh image is dumped containing the updated metadata. This work has provided opportunities to make various other improvements to Consfigurator, especially with regard to dumping and reinvoking images. Making SBCL capable of entering user namespaces required a change upstream, which made it into the recent SBCL 2.1.8 release. I m very grateful to the SBCL developers for their engagement with my project. I ve been able to add a workaround so that Consfigurator can still enter user namespaces when run on the version of SBCL included in Debian stable. I also discovered that deploying all of my laptop, lxctest and lxc1 at once generates enough output to fill up a pipe, thus revealing a deadlock in Consfigurator s IPC, which it was good to become aware of and fix. That involved writing my first multi-threaded Lisp, as there are two pipes that need to be kept from filling up, and to my surprise it worked first time. Take that Haskell

Wasted quite some hours until I found a working Modeline in this stack exchange post so the ThinkPad works with a HDMI attached Samsung QHD display. Internal display of the ThinkPad is a FHD display detected as eDP-1, the external one is DP-3 and according to the packaging known by Samsung as S24A600NWU. The auto deteced EDID modes for QHD - 2560x1440 - did not work at all, the display simply stays dark. After a lot of back and forth with the i915 driver vs nouveau vs nvidia/nvidia-drm with and without modesetting, the following Modeline did the magic:

xrandr --newmode 2560x1440_54.97  221.00  2560 2608 2640 2720  1440 1443 1447 1478  +HSync -VSync
xrandr --addmode DP-3 2560x1440_54.97
xrandr --output DP-3 --mode 2560x1440_54.97 --right-of eDP-1 --primary

Modelines for 50Hz and 60Hz generated with cvt 2560 1440 60 did not work, neither did the one extracted with edid-decode -X from the hex blob found in .local/share/xorg/Xorg.0.log. From the auto-detected Modelines FHD - 1920x1080 - did work. In case someone struggles with a similar setup, that might be a starting point. Fun part, if I attach my several years old Dell E7470 everything is just fine out of the box. But that one just has an Intel GPU and not the unholy combination I've here:

$ lspci grep -E "VGA 3D"
00:02.0 VGA compatible controller: Intel Corporation CometLake-H GT2 [UHD Graphics] (rev 05)
01:00.0 3D controller: NVIDIA Corporation GP107GLM [Quadro P620] (rev ff)

Some time ago I looked briefly at an Envertech data logger for small scale photovoltaic setups. Turned out that PV inverter are kinda unreliable, and you really have to monitor them to notice downtimes and defects. Since my pal shot for a quick win I've cobbled together another Python script to query the portal at www.envertecportal.com, and report back if the generated power is down to 0. The script is currently run on a vserver via cron and reports back via the system MTA. So yeah, you need to have something like that already at hand. Script and Configuration You've to provide your PV systems location with latitude and longitude so the script can calculate (via python3-suntime) the sunrise and sunset times. At the location we deal with we expect to generate some power at least from sunrise + 1h to sunet - 1h. That is tunable via the configuration option toleranceSeconds. Retrieving the stationId is a bit ugly because it's not provided via any API, instead it's rendered serverside into the website. So I just logged in on the portal and picked it up by looking into the page source. www.envertecportal.com API I guess this is some classic in the IoT land, but neither the documentation provided on the portal frontpage as docx, nor the API docs at port 8090 are complete and correct. The few bits I gathered via the Firefox Web Developer Tools are:

1. Login https://www.envertecportal.com/apiaccount/login - POST, sent userName and pwd containing your login name and password. The response JSON is very explicit if your login was not successful and why.
2. Store the session cookie called ASP.NET_SessionId for use on all subsequent requests.
3. Retrieve station info https://www.envertecportal.com/ApiStations/getStationInfo - POST, sent ASP.NET_SessionId and stationId with the ID of the station. Returns a JSON with an object named Data. The field Power contains the currently generated power as a float with two digits (e.g. 0.01).
4. Logout https://www.envertecportal.com/apiAccount/Logout - POST, sent ASP.NET_SessionId.

Some Surprises There were a few surprises, maybe they help others dealing with an Envertech setup.

1. The portal truncates passwords at 16 chars.
2. The "Forget Password?" function mails you back the password in plain text (that's how I learned about 1.).
3. The login API endpoint reporting the exact reason why the login failed is somewhat out of fashion. Though this one is probably not a credential stuffing target because there is no money to make, so don't care.
4. The data logger reports the data to www.envertecportal.com at port 10013.
5. There is some checksuming done on the reported data, but the system is not replay safe. So you can sent it any valid data string at a later time and get wrong data recorded.
6. People at forum.fhem.de decoded some values but could not figure out the checksuming so far.

Review: Black Sun, by Rebecca Roanhorse

Series:	Between Earth and Sky #1
Publisher:	Saga Press
Copyright:	October 2020
ISBN:	1-5344-3769-X
Format:	Kindle
Pages:	454

Serapio has been crafted and trained to be the vessel for a god. He grew up in Obregi land, far from his ancestral home, but he will return to Tova at the appropriate time and carry the hopes of the Carrion Crow clan with him. Xiala is a ship captain, a woman, and a Teek. That means she's a target. Teek have magic, which makes them uncanny and dangerous. They're also said to carry that magic in their bones, which makes them valuable in ways that are not pleasant for the Teek. Running afoul of the moral codes of Cuecola is therefore even more dangerous to her than it would be to others, which is why she accepts a bargain to run errands for a local lord for twelve years, paid at the end of that time with ownership of a ship and crew. The first task: ferry a strange man to the city of Tova. Meanwhile, in Tova, the priestess Naranpa has clawed her way to the top of the Sky Made hierarchy from an inauspicious beginning in the poor district of Coyote's Maw. She's ruthlessly separated herself from her despised beginnings and focused her attention on calming Tova in advance of the convergence, a rare astronomical alignment at the same time as the winter solstice. But Carrion Crow holds a deep-seated grudge at their slaughter by the priesthood during the Night of Knives, and Naranpa's position atop the religious order that partly rules Tova's fractious politics is more precarious than she thinks. I am delighted that more fantasy is drawing on mythologies and histories other than the genre default of western European. It's long overdue for numerous reasons and a trend to be rewarded. But do authors writing fantasy in English who reach for Mesoamerican cultures have to gleefully embrace the excuse to add more torture? I'm developing an aversion to this setting (which I do not want to do!) because every book seems to feature human sacrifice, dismemberment, or some other horror show. Roanhorse at least does not fill the book with that (there's lingering child abuse but nothing as sickening as the first chapter), but that makes the authorial choice to make the torture one's first impression of this book even odder. Our introduction to Serapio is a scene that I would have preferred to have never read, and I don't think it even adds much to the plot. Huge warnings for people who don't want to read about a mother torturing her son, or about eyes in that context. Once past that introduction, Black Sun settles into a two-thread fantasy, one following Xiala and Serapio's sea voyage and the other following Naranpa and the political machinations in Tova. Both the magic systems and the political systems are different enough to be refreshing, and there are a few bits of world-building I enjoyed (a city built on top of rocks separated by deep canyons and connected with bridges, giant intelligent riding crows, everything about the Teek). My problem was that I didn't care what happened to any of the characters. Naranpa spends most of the book dithering and whining despite a backstory that should have promised more dynamic and decisive responses. The other character from Tova introduced somewhat later in the book is clearly "character whose story will appear in the next volume"; here, he's just station-keeping and representing the status quo. And while it's realistic given the plot that Serapio is an abused sociopath, that didn't mean I enjoyed reading his viewpoint or his childhood abuse. Xiala is the best character in the book by far and I was warming to the careful work she has to do to win over an unknown crew, but apparently Roanhorse was not interested in that. Instead, the focus of Xiala's characterization turns to a bad-boy romance that did absolutely nothing for me. This will be a matter of personal taste; I know this is a plot feature for many readers. But it had me rolling my eyes and turning the pages to get to something more interesting (which, sadly, was not forthcoming). It also plays heavily on magical disabled person cliches, like the blind man being the best fighter anyone has met. I did not enjoy this book very much, but there were some neat bits of world-building and I could see why other people might disagree. What pushed me into actively recommending against it (at least for now) is the publishing structure. This is the first book of a trilogy, so one can expect the major plot to not be resolved by the end of the book. But part of the contract with the reader when publishing a book series is that each volume should reach some sense of closure and catharsis. There will be cliffhangers and unanswered questions, but there should also be enough plot lines that are satisfactorily resolved to warrant publishing a book as a separate novel. There is none of that here. This is the first half (or third) of a novel. It introduces a bunch of plot lines, pulls them together, describes an intermediate crisis, and then simply stops. Not a single plot line is resolved. This is made worse by the fact this series (presumably, as I have only seen the first book) has a U-shaped plot: everything gets worse and worse until some point of crisis, and then presumably the protagonists will get their shit together and things will start to improve. I have soured on U-shaped plots since the first half of the story often feels like a tedious grind (eat your vegetables and then you can have dessert), but it's made much worse by cutting the book off at the bottom of the U. You get a volume, like Black Sun, that's all setup and horror and collapse, with no payoff or optimism. After two tries, I have concluded that Roanhorse is not for me. This is clearly a me problem rather than a Roanhorse problem, given how many other people love both Black Sun and her Sixth World series, but this is the second book of hers where I mildly enjoyed the world building but didn't care about any of the characters. Ah well, tastes will differ. Even if you get along with Roanhorse, though, I recommend against starting this book until the second half of it is published (currently scheduled for 2022). As it stands, it's a wholly unsatisfying reading experience. Followed by the not-yet-published Fevered Star. Rating: 4 out of 10

So today is the day when a new Debian release comes out! Congratulations to everybody, and thanks a lot mainly to the Release Team. Lots of very hard work was put into making Debian 11 Bullseye a reality! My very personal way to celebrate this was to do a somewhat different Debian install at home. Why different? Well, I have quite a bit of old, older and frankly elderly laptops at home. And as many of you know, I have done more than my fair share of Raspberry Pi installs I have played and worked with assorted ARM machines at least since 2013, and I cannot consider myself a newbie with them by any means. But this is the first time I installed Debian on a mass-market, decently-specced ARM64-based laptop. Yes, I know the Pinebook has been there like for ages, but it really does feel like a computer to show off and not to use seriously (and I ve seen probably too many people fiddling with it, unable to get $foo to work). So I got myself a used Lenovo Yoga C630. Yes, a discontinued product it seems Lenovo was not able to properly market this machine, and it had a pretty short shelf life the machine was available for samples in late 2018 and for general sale in 2019! The specs are quite decent:

• Snapdraon SDM850 AArch64 8-core CPU. Runs at frequencies between 300 and 2840 MHz.
• 8GB RAM
• 128GB SSD drive
• FHD (1920x1080) 13 screen
• 1.2Kg weight
• Battery life! The supposed capacity of a brand-new system is 22 hours. I don t expect a second-hand computer to achieve that. But I ve been using the system for ~10hr today without a connection, and it still has 22.5% battery!

Installing it via an almost-standard debian-installer is almost straightforward does require the installer to know what he is doing but is not too different from a regular Debian install. The AArch64 laptops project has done quite a feat in getting a d-i image ready to be inserted as a USB drive, and comprehensive instructions to help through the process. The shipped scripts even reap the Windows partition for the firmware images! I have reduced Windows to 25GB, but having only a 128GB drive, it still is a little bit too much.. I guess I ll blow it away sooner rather than later. The installer image has a regular GNOME install, which works beautifully, but I promptly replaced it with i3, as it s fundamental for me to work happily. Of course, the computer has quirks, more than I d expect from a regular x86 system, but well within what I expected to achieve during my first day with it. The issues I have most noted are:

• No HDMI support via the USB-C displayport. While I don t expect to go to conferences or even classes in the next several months, I hope this can be fixed before I do. It s a potential important issue for me.
• I haven t been able to get the sound working. Steev (thanks a lot for all the info and hardware you sent my way!) prepared a 5.12 kernel package that should result in working sound. It hasn t worked. I haven t really debugged it, of course. Sound now works! I was missing the loading of the relevant DTBs. It was just a matter of running /usr/local/bin/install-dtbs.py /usr/lib/linux-image-5.12.19-custom/ /boot/efi/dtb. And this should remain handy, as I ll have to do so for every new kernel I install (hint, hint: It should be called from /etc/kernel/postinst.d or something like that. I should check and propose a patch!)
• Fingerprint reader. Just a novelty I don t really need; my fingers know how to quickly type a long-ish password. But it s a cool item (-:
• I have seen some seemingly random cases where the trackpad freezes after coming back from suspension. Touchscreen mouse support still works (but, of course, sucks and should be used only for backup). Will definitively look into it.
• There are some cases where i3status panics and dies. i3status is a package as simple as can be, so I ll also definitively try and debug what is going wrong. Switched away from i3status (details follow )
• The video output is quite slow. The default install used Wayland, which gets in the way of many of my needed use cases (mainly, screen grabbing/sharing), so I switched to xserer-xorg; according to the logs, I m using the framebuffer device. Will later see if there is support for the Qualcomm Adreno 630GPU. Still, it manages to run some OpenGL xscreensaver hacks at 30FPS, ~40% load But urxvt is dismally slow. Emilio prompted me to give Wayland a shot, specially in this kind of hardware. While I still have to wrap my head around some of the changes it introduces, I am mostly there. I m using Sway instead of my good ol i3, which I love under X, and it feels very much at home. I have some re-learning to do at the stitches between protocols.

Of course, more quirks will surely appear with use. And I ll start trying to address some of them. So Happy Bullseye! Happy Debian 11! Enjoy a great release! \o/

Running a playbook in a remote chroot or container is not supported by Ansible, but I have invented a good workaround to do it anyway. The first step is to install Mitogen for Ansible (ansible-mitogen in Debian) and then configure ansible.cfg to use it:

[defaults]
strategy = mitogen_linear

But everybody should use Mitogen anyway, because it makes Ansible much faster. The trick to have Ansible operate in a chroot is to make it call a wrapper script instead of Python. The wrapper can be created manually or by another playbook, e.g.:

  vars:
  - fsroot: /mnt
  tasks:
  - name: Create the chroot wrapper
    copy:
      dest: "/usr/local/sbin/chroot_ inventory_hostname_short "
      mode: 0755
      content:  
        #!/bin/sh -e
        exec chroot  fsroot  /usr/bin/python3 "$@"
  - name: Continue with stage 2 inside the chroot
    debug:
      msg:
        - "Please run:"
        - "ansible-playbook therealplaybook.yaml -l  inventory_hostname  -e ansible_python_interpreter=/usr/local/sbin/chroot_ inventory_hostname_short "

This works thanks to Mitogen, which funnels all remote tasks inside that single call to Python. It would not work with standard Ansible, because it copies files to the remote system with SFTP and would do it outside of the chroot. The same principle can also be applied to containers by changing wrapper script, e.g:

#!/bin/sh -e
exec systemd-run --quiet --pipe --machine= container_name  --service-type=exec /usr/bin/python3 "$@"

After the wrapper will have been installed then you can run the real playbook by setting the ansible_python_interpreter variable, either on the command line, in the inventory or anywhere else that variables can be defined:

ansible-playbook therealplaybook.yaml -l  inventory_hostname  -e ansible_python_interpreter=/usr/local/sbin/chroot_ inventory_hostname_short 

BBI Kenya and live Supreme Court streaming on YT The last few weeks have been unrelenting as all sorts of news have been coming in, mostly about the downturn in the Economy, Islamophobia in India on the rise, Covid, and electioneering. However, in the last few days, Kenya surpassed India in live-streaming proceeds in a Court of Appeals about BBI or Building Bridges Initiative. A background filler article on the topic can be found in BBC. The live-streaming was done via YT and if wants to they can start from

https://www.youtube.com/watch?v=JIQzpmVKvro One can also subscribe to K24TV which took the initiative of sharing the proceedings with people worldwide. If K24TV continues to share SC proceedings of Kenya, that would add to the soft power of Kenya. I will not go into the details of the case as Gautam Bhatia who has been following the goings-on in Kenya is a far better authority on the subject. In fact, just recently he shared about another Kenyan judgment from a trial which can be seen here. He has shared the proceedings and some hot takes on the Twitter thread started by him. Probably after a couple of weeks or more when he has processed what all has happened there, he may also share some nuances although many of his thoughts would probably go to his book on Comparative Constitutional Law which he hopes to publish maybe in 2021/2022 or whenever he can. Such televised proceedings are sure to alleviate the standing of Kenya internationally. There has been a proposal to do similar broadcasts by India but with surveillance built-in, so they know who is watching. The problems with the architecture and the surveillance built-in have been shared by Srinivas Kodali or DigitalDutta quite a few times, but that probably is a story for another day.

Uttar Pradesh Population Control Bill

Hindus comprise 83% of Indian couples with more than two child children

The U.P. Population Bill came and it came with lot of prejudices. One of the prejudices is the idea that Muslims create or procreate to have the most children. Even with data is presented as shared above from NFHS National Family Health Survey which is supposed to carry our surveys every few years did the last one around 4 years back. The analysis from it has been instrumental not only in preparing graphs as above but also sharing about what sort of death toll must have been in rural India. And as somebody who have had the opportunity in the past, can vouch that you need to be extremely lucky if something happens to you when you are in a rural area. Even in places like Bodh Gaya (have been there) where millions of tourists come as it is one of the places not to be missed on the Buddhism tourist circuit, the medical facilities are pretty underwhelming. I am not citing it simply because there are too many such newspaper reports from even before the pandemic, and both the State and the Central Govt. response has been dismal. Just a few months back, they were recalled. There were reports of votes being bought at INR 1000/- (around $14) and a bottle or two of liquor. There used to be a time when election monitoring whether national or state used to be a thing, and you had LTO s (Long-time Observers) and STO s (Short-Term Observers) to make sure that the election has been neutral. This has been on the decline in this regime, but that probably is for another time altogether. Although, have to point out the article which I had shared a few months ago on the private healthcare model is flawed especially for rural areas. Instead of going for cheap, telemedicine centers that run some version of a Linux distro. And can provide a variety of services, I know Kerala and Tamil Nadu from South India have experimented in past but such engagements need to be scaled up. This probably will come to know when the next time I visit those places (sadly due to the virus, not anytime soonish.:( ) . Going back to the original topic, though, I had shared Hans Rosling s famous Ted talk on population growth which shows that even countries which we would not normally associate with family planning for e.g. the middle-east and Africa have also been falling quite rapidly. Of course, when people have deeply held prejudices, then it is difficult. Even when sharing China as to how they had to let go of their old policy in 2016 as they had the thing for leftover men . I also shared the powerful movie So Long my Son. I even shared how in Haryana women were and are trafficked and have been an issue for centuries but as neither suits the RW propaganda, they simply refuse to engage. They are more repulsed by people who publish this news rather than those who are actually practicing it, as that is culture . There is also teenage pregnancy, female infanticide, sex-selective abortion, etc., etc. It is just all too horrible to contemplate. Personal anecdote I know a couple, or they used to be a couple, where the gentleman wanted to have a male child. It was only after they got an autistic child, they got their DNA tested and came to know that the gentleman had a genetic problem. He again forced and had another child, and that too turned out to be autistic. Finally, he left the wife and the children, divorced them and lived with another woman. Almost a decade of the wife s life was ruined. The wife before marriage was a gifted programmer employed at IBM. This was an arranged marriage. After this, if you are thinking of marrying, apart from doing astrology charts, also look up DNA compatibility charts. Far better than ruining yours or the women s life. Both the children whom I loved are now in heaven, god bless them If one wants to, one can read a bit more about the Uttar Pradesh Population bill here. The sad part is that the systems which need fixing, nobody wants to fix. The reason being simple. If you get good health service by public sector, who will go to the private sector. In Europe, AFAIK they have the best medical bang for the money. Even the U.S. looks at Europe and hopes it had the systems that Europe has but that again is probably for another day.

South Africa and India long-lost brothers. As had shared before, after the 2016 South African Debconf convention, I had been following South Africa. I was happy when FeesMustFall worked and in 2017 the then ANC president Zuma declared it in late 2017. I am sure that people who have been regular visitors to this blog know how my position is on student loans. They also must be knowing that even in U.S. till the 1970s it had free education all the way to be a lawyer and getting a lawyer license. It is only when people like Thurgood Marshall, Martin Luther King Jr., and others from the civil rights movement came out as a major force that the capitalists started imposing fees. They wanted people who could be sold to corporate slavery, and they won. Just last week, Biden took some steps and canceled student loans and is working on steps towards broad debt forgiveness. Interestingly, NASA has an affirmative diversity program for people from diverse backgrounds, where a couple of UC (Upper Caste) women got the job. While they got the job, the RW (Right-Wing) was overjoyed as they got jobs on merit . Later, it was found that both the women were the third or fourth generation of immigrants in U.S.

NASA Federal Equal Opportunity Policy Directive NPD 3713 2H

Going back to the original question and topic, while there has been a concerning spate of violence, some calling it the worst sort of violence not witnessed since 1994. The problem, as ascertained in that article, is the same as here in India or elsewhere. Those, again, who have been on my blog know that merit 90% of the time is a function of privilege and there is a vast amount of academic literature which supports that. If, for a moment, you look at the data that is shared in the graph above which shows that 83% of Hindus and 13% of Muslims have more than 2 children, what does it show, it shows that 83+13 = 96% of the population is living in insecurity. The 5% are the ones who have actually consolidated more power during this regime rule in India. Similarly, from what I understood living in Cape Town for about a month, it is the Dutch Afrikaans as they like to call themselves and the immigrants who come from abroad who have enjoyed the fruits of tourism and money and power while the rest of the country is dying due to poverty. It is the same there, it is the same here. Corruption is also rampant in both countries, and the judiciary is virtually absent from both communities in India and SA. Interestingly, South Africa and India have been at loggerheads, but I suspect that is more due to the money and lobbying power by the Dutch. Usually, those who have money power, do get laws and even press on their side, and it is usually the ruling party in power. I cannot help but share about the Gupta brothers and their corruption as I came to know about it in 2016. And as have shared that I m related to Gupta s on my mother s side, not those specific ones but Gupta as a clan. The history of the Gupta dynasty does go back to the 3rd-4th century. Equally interesting have been Sonali Ranade s series of articles which she wrote in National Herald, the latest on exports which is actually the key to taking India out of poverty rather than anything else. While in other countries Exporters are given all sort of subsidies, here it is being worked as how to give them less. This was in Economic times hardly a week back

Export incentive schemes being reduced

I can t imagine the incredible stupidity done by the Finance Minister. And then in an attempt to prove that, they will attempt to present a rosy picture with numbers that have nothing to do with reality. Interestingly enough, India at one time was a major exporter of apples, especially from Kashmir. Now instead of exporting, we are importing them from Afghanistan as well as Belgium and now even from the UK. Those who might not want to use the Twitter link could use this article. Of course, what India got out of this trade deal is not known. One can see that the UK got the better deal from this. Instead of investing in our own capacity expansion, we are investing in increasing the capacity of others. This is at the time when due to fuel price hike (Central taxes 66%) demand is completely flat. And this is when our own CEA (Chief Economic Adviser) tells us that growth will be at the most 6-7% and that too in 2023-2024 while currently, the inflation rate is around 12%. Is it then any wonder that almost 70% are living on Govt. ration and people in the streets of Kolkata, Assam, and other places have to sell kidneys to make sure they have some money for their kids for tomorrow. Now I have nothing against the UK but trade negotiation is an art. Sadly, this has been going on for the last few years. The politicians in India fool the public by always telling of future trade deals. Sadly, as any businessman knows, once you have compromised, you always have to compromise. And the more you compromise, the more you weaken the hand for any future trade deals.

IIT pupil tries to sell kidney to repay loan, but no takers for Dalit organ.

The above was from yesterday s Times of India. Just goes to show how much people are suffering. There have been reports in vernacular papers of quite a few people from across regions and communities are doing this so they can live without pain a bit. Almost all the time, the politicians are saved as only few understand international trade, the diplomacy and the surrounding geopolitics around it. And this sadly, is as much to do with basic education as much as it is to any other factor

Suli Deals About a month back on the holy day of Ramzan or Ramadan as it is known in the west, which is beloved by Muslims, a couple of Muslim women were targeted and virtually auctioned. Soon, there was a flood and a GitHub repository was created where hundreds of Muslim women, especially those who have a voice and fearlessly talk about their understanding about issues and things, were being virtually auctioned. One week after the FIR was put up, to date none of the people mentioned in the FIR have been arrested. In fact, just yesterday, there was an open letter which was published by livelaw. I have saved a copy on WordPress just in case something does go wrong. Other than the disgust we feel, can t say much as no action being taken by GOI and police.

IT Rules 2021 and Big Media After almost a year of sleeping when most activists were screaming hoarsely about how the new IT rules are dangerous for one and all, big media finally woke up a few weeks back and listed a writ petition in Madras High Court of the same. Although to be frank, the real writ petition was filed In February 2021, classical singer, performer T.M. Krishna in Madras High Court. Again, a copy of the writ petition, I have hosted on WordPress. On 23rd June 2021, a group of 13 media outlets and a journalist have challenged the IT Rules, 2021. The Contention came from Digital News Publishers Association which is made up of the following news companies: ABP Network Private Limited, Amar Ujala Limited, DB Corp Limited, Express Network Pvt Ltd, HT Digital Streams Limited, IE Online Media Services Pvt Ltd, Jagran Prakashan Limited, Lokmat Media Private Limited, NDTV Convergence Limited, TV Today Network Limited, The Malayala Manorama Co (P) Ltd, Times Internet Limited, and Ushodaya Enterprises Private Limited. All the above are heavyweights in the markets where they operate. The reason being simple, when these media organizations came into being, the idea was to have self-regulation, which by and large has worked. Now, the present Govt. wants each news item to be okayed by them before publication. This is nothing but blatant misuse of power and an attempt at censorship. In fact, the Tamil Nadu BJP president himself made a promise of the same. And of course, what is true and what is a lie, only GOI knows and will decide for the rest of the country. If somebody remembers Joseph Goebbels at this stage, it is merely a coincidence. Anyways, 3 days ago Supreme Court on 14th July the Honorable Supreme Court asked the Madras High Court to transfer all the petitions to SC. This, the Madras High Court denied as cited/shared by Meera Emmanuel, a reporter who works with barandbench. The Court says nothing doing, let this happen and then the SC can entertain the motion of doing it that level. At the same time, they would have the benefit of Madras High Court opinion as well. It gave the center two weeks to file a reply. So, either of end-week of July or latest by August first week, we might be able to read the Center s reply on the same. The SC could do a forceful intervention, but it would lead to similar outrage as has been witnessed in the past when a judge commented that if the SC has to do it all, then why do we need the High Courts, district courts etc. let all the solutions come from SC itself. This was, admittedly, frustration on the part of the judge, but due in part to the needless intervention of SC time and time again. But the concerns had been felt around all the different courts in the country.

Sedition Law A couple of days ago, the Supreme Court under the guidance of Honorable CJI NV Ramanna, entertained the PIL filed by Maj Gen S G Vombatkere (Retd.) which asked simply that the sedition law which was used in the colonial times by the British to quell dissent by Mahatma Gandhi and Bal Gangadhar Tilak during the Indian freedom struggle. A good background filler article can be found on MSN which tells about some recent cases but more importantly how historically the sedition law was used to quell dissent during India s Independence. Another article on MSN actually elaborates on the PIL filed by Maj Gen S. G. Vombatkere. Another article on MSN tells how sedition law has been challenged and changed in 10 odd countries. I find it equally sad and equally hilarious that the Indian media whose job is to share news and opinion on this topic is being instead of being shared more by MSN. Although, I would be bereft of my duty if I did not share the editorial on the same topic by the Hindu and Deccan Chronicle. Also, an interesting question to ask is, are there only 10 countries in the world that have sedition laws? AFAIK, there are roughly 200 odd countries as recognized by WTO. If 190 odd countries do not have sedition laws, it also tells a lot about them and a lot about the remaining 10. Also, it came to light that police are still filing laws under sec66A which was declared null and void a few years ago. It was replaced with section 124A if memory serves right and it has more checks and balances.

Danish Siddiqui, Pulitzer award-winning and death in Afghanistan Before I start with Danish Siddiqui, let me share an anecdote that I think I have shared on the blog years ago about how photojournalists are. Again, those who know me and those who follow me know how much I am mad both about trains and planes (civil aviation). A few months back, I had shared a blog post about some of the biggest railway systems in the world which shows that privatization of Railways doesn t necessarily lead to up-gradation of services but definitely leads to an increase in tariff/fares. Just had a conversation couple of days ago on Twitter and realized that need to also put a blog post about civil aviation in India and the problems it faces, but I digress. This was about a gentleman who wanted to take a photo of a particular train coming out of a valley at a certain tunnel at two different heights, one from below and one from above the train. This was several years ago, and while I did share that award-winning photograph then, it probably would take me quite a bit of time and effort to again look it up on my blog and share. The logistics though were far more interesting and intricate than I had first even thought of. We came around a couple of days before the train was supposed to pass that tunnel and the valley. More than half a dozen or maybe more shots were taken throughout the day by the cameras. The idea was to see how much light was being captured by the cameras and how much exposure was to be given so that the picture isn t whitened out or is too black. Weather is the strangest of foes for a photojournalist or even photographers, and the more you are in nature, the more unpredictable it is and can be. We were also at a certain height, so care had to be taken in case light rainfall happens or dew falls, both not good for digital cameras. And dew is something which will happen regardless of what you want. So while the two days our gentleman cameraman fiddled with the settings to figure out correct exposure settings, we had one other gentleman who was supposed to take the train from an earlier station and apprise us if the train was late or not. The most ideal time would be at 0600 hrs. When the train would enter the tunnel and come out and the mixture of early morning sun rays, dew, the flowers in the valley, and the train would give a beautiful effect. We could stretch it to maybe 0700 hrs. Anything after that would just be useless, as it wouldn t have the same effect. And of all this depended on nature. If the skies were to remain too dark, nothing we could do about it, if the dewdrops didn t fall it would all be over. On the day of the shoot, we were told by our compatriot that the train was late by half an hour. We sank a little on hearing that news. Although Photoshop and others can do touch-ups, most professionals like to take as authentic a snap as possible. Everything had been set up to perfection. The wide-angle lenses on both the cameras with protections were set up. The tension you could cut with a knife. While we had a light breakfast, I took a bit more and went in the woods to shit and basically not be there. This was too tensed up for me. Returned an hour to find everybody in a good mood. Apparently, the shoot went well. One of the two captured it for good enough. Now, this is and was in a benign environment where the only foe was the environment. A bad shot would have meant another week in the valley, something which I was not looking forward to. Those who have lived with photographers and photojournalists know how self-involved they can be in their craft, while how grumpy they can be if they had a bad shoot. For those, who don t know, it is challenging to be friends with such people for a long time. I wish they would scream more at nature and let out the frustrations they have after a bad shoot. But again, this is in a very safe environment. Now let s cut to Danish Siddiqui and the kind of photojournalism he followed. He followed a much more riskier sort of photojournalism than the one described above. Krittivas Mukherjee in his Twitter thread shared how reporters in most advanced countries are trained in multiple areas, from risk assessment to how to behave in case you are kidnapped, are in riots, hostage situations, etc. They are also trained in all sorts of medical training from treating gunshot wounds, CPR, and other survival methods. They are supposed to carry medical equipment along with their photography equipment. Sadly, these concepts are unknown in India. And even then they get killed. Sadly, he attributes his death to the thrill of taking an exclusive photograph. And the gentleman s bio reads that he is a diplomat. Talk about tone-deafness On another completely different level was Karen Hao who was full of empathy as she shared the humility, grace, warmth and kinship she describes in her interaction with the photojournalist. His body of work can be seen via his ted talk in 2020 where he shared a brief collage of his works. Latest, though in a turnaround, the Taliban have claimed no involvement in the death of photojournalist Danish Siddiqui. This could be in part to show the Taliban in a more favorable light as they do and would want to be showcased as progressive, even though they are forcing that all women within a certain age become concubines or marry the fighters and killing the minority Hazaras or doing vile deeds with them. Meanwhile, statements made by Hillary Clinton almost a decade, 12 years ago have come back into circulation which stated how the U.S. itself created the Taliban to thwart the Soviet Union and once that job was finished, forgot all about it. And then in 2001, it landed back in Afghanistan while the real terrorists were Saudi. To date, not all documents of 9/11 are in the public domain. One can find more information of the same here. This is gonna take probably another few years before Saudi Arabia s whole role in the September 11 attacks will be known. Last but not the least, came to know about the Pegasus spyware and how many prominent people in some nations were targeted, including in mine India. Will not talk more as it s already a big blog post and Pegasus revelations need an article on its own.

Tonight I m provisioning a new virtual machine at Hetzner and I wanted to share how Consfigurator is helping with that. Hetzner have a Debian buster image you can start with, as you d expect, but it comes with things like cloud-init, preconfiguration to use Hetzner s apt mirror which doesn t serve source packages(!), and perhaps other things I haven t discovered. It s a fine place to begin, but I want all the configuration for this server to be explicit in my Consfigurator consfig, so it is good to start with pristine upstream Debian. I could boot one of Hetzner s installation ISOs but that s slow and manual. Consfigurator can replace the OS in the VM s root filesystem and reboot for me, and we re ready to go. Here s the configuration:

(defhost foo.silentflame.com (:deploy ((:ssh :user "root") :sbcl))
  (os:debian-stable "buster" :amd64)
  ;; Hetzner's Debian 10 image comes with a three-partition layout and boots
  ;; with traditional BIOS.
  (disk:has-volumes
   (physical-disk
    :device-file "/dev/sda" :boots-with '(grub:grub :target "i386-pc")))
  (on-change (installer:cleanly-installed-once
              nil
              ;; This is a specification of the OS Hetzner's image has, so
              ;; Consfigurator knows how to install SBCL and debootstrap(8).
              ;; In this case it's the same Debian release as the replacement.
              '(os:debian-stable "buster" :amd64))
    ;; Clear out the old OS's EFI system partition contents, in case we can
    ;; switch to booting with EFI at some point (if we wanted we could specify
    ;; an additional x86_64-efi target above, and grub-install would get run
    ;; to repopulate /boot/efi, but I don't think Hetzner can boot from it yet).
    (file:directory-does-not-exist "/boot/efi/EFI")
    (apt:installed "linux-image-amd64")
    (installer:bootloaders-installed)
    (fstab:entries-for-volumes
     (disk:volumes
       (mounted-ext4-filesystem :mount-point "/")
       (partition
        (mounted-fat32-filesystem
         :mount-options '("umask=0077") :mount-point "/boot/efi"))))
    (file:lacks-lines "/etc/fstab" "# UNCONFIGURED FSTAB FOR BASE SYSTEM")
    (file:is-copy-of "/etc/resolv.conf" "/old-os/etc/resolv.conf")
    (mount:unmounted-below-and-removed "/old-os"))
  (apt:mirror "http://ftp.de.debian.org/debian")
  (apt:no-pdiffs)
  (apt:standard-sources.list)
  (sshd:installed)
  (as "root" (ssh:authorized-keys +spwsshkey+))
  (sshd:no-passwords)
  (timezone:configured "Etc/UTC")
  (swap:has-swap-file "2G")
  (network:clean-/etc/network/interfaces)
  (network:static "enp1s0" "xxx.xxx.xxx.xxx" "xxx.xxx.1.1" "255.255.255.255"))

and to use it you evaluate this at the REPL:

CONSFIG> (deploy ((:ssh :user "root" :hop "xxx.xxx.xxx.xxx") :sbcl) foo.silentflame.com)

Here the :HOP parameter specifies the IP address of the new machine, as DNS hasn t been updated yet. Consfigurator installs SBCL and debootstrap(8), prepares a minimal system, replaces the contents of /, gets to work applying the other properties, and then reboots. This gets us a properly populated fstab:

UUID=...            /           ext4    relatime    0   1
PARTUUID=...        /boot/efi   vfat    umask=0077  0   2
/var/lib/swapfile   swap        swap    defaults    0   0

(slightly doctored for more readable alignment) There s ordering logic so that the swapfile will end up after whatever filesystem contains it; a UUID is used for ext4 filesystems, but for fat32 filesystems, to be safe, a PARTUUID is used. The application of (INSTALLER:BOOTLOADERS-INSTALLED) handles calling both update-grub(8) and grub-install(8), relying on the metadata specified about /dev/sda. Next time we execute Consfigurator against the machine, it ll ignore all the property applications attached to the application of (INSTALLER:CLEANLY-INSTALLED-ONCE) with ON-CHANGE, and just apply everything following that block. There are a few things I don t have good solutions for. When you boot Hetzner s image the primary network interface is eth0, but then for a freshly debootstrapped Debian you get enp1s0, and I haven t got a good way of knowing what it ll be (if you know it ll have the same name, you can use (NETWORK:PRESERVE-STATIC-ONCE) to create a file in /etc/network/interfaces.d based on the current default route and corresponding interface). Another tricky thing is SSH host keys. It s easy to use Consfigurator to add host keys to your laptop s ~/.ssh/known_hosts, but in this case the host key changes back and forth from whatever the Hetzner image has and the newly generated key you get afterwards. One option might be to copy the old host keys out of /old-os before it gets deleted, like how /etc/resolv.conf is copied. This work is based on Propellor s equivalent functionality. I think my approach to handling /etc/fstab and bootloader installation is an improvement on what Joey does.

MIT Technology Review has an interesting article about Google Project Zero shutting down a western intelligence operation [1]. There s an Internet trend of people eating rotten meat they call high meat (rotten meat) [2]. This is up there with people setting themselves on fire and nut shot videos. A young female who was making popular Twitter posts about motorbikes turned out to be a 50yo man using deep fake technology [3]. He has long hair IRL and just needed to replace his face. After coming out of the closet he has continued making such videos and remains popular. FYHTECH has an informative blog post about using sgdisk to backup and restore GPT partition tables [4]. This is in the Debian package gdisk along with several other tools for managing partition tables. One interesting thing to note is that you can backup a partition table and restore to a smaller device (with a bunch of warnings that you can ignore if you know what you are doing). This is the only way I ve discovered to cleanly truncate a GPT partitioned disk, which is sometimes necessary when running VMs. Insightful blog post about PCIe bifurcation and how PCIe lanes are assigned to sockets [5]. This explains why many motherboards have sockets with unused PCIe lanes, EG *8 sockets that are wired for *4. The PCIe slots all go back to the CPU which has a limited number of *16 PCIe connections that are bifurcated to make the larger number of PCIe slots on the motherboard. New Republic has an interesting article on the infamous transphobe Jordan Peterson s battle with tranquiliser dependency [6]. Wired has an interesting article about the hack of RSA infrastructure related to the SecureID keys 10 years ago [7]. Apparently some 10 year NDAs had delayed it. There are many posts about the situation with Freenode, I think that this one best captures the problems in the shortest amount of text [8]. You could spend a few hours reading about it as I have just done, but just reading this gives you the basics that you need to know to avoid Freenode. That blog post has links to articles about Andrew Lee s involvement with Mt Gox and claims to be the heir to the throne of Korea (which is not a monarchy). Nicholas Wade wrote an insightful and informative article about the origin of Covid19, which leads to the conclusion that it was made in a Chinese laboratory [9]. I first saw this in David Brin s Facebook feed. I would be hesitant to share this sort of thing if it wasn t reviewed by a reliable source, I think David Brin has the skill to analyse this sort of article and the contacts to allow him to seek verification of any scientific issues that are outside his field. I believe that this article is reliable and it s conclusion is most likely to be correct. Interesting Wired article about an art project using display computers at Apple stores to photograph people [10]. Ends with a visit from the FBI.

• [1] https://tinyurl.com/yhoxy6lf
• [2] https://tinyurl.com/ydtssdtx
• [3] https://tinyurl.com/ygwwyrww
• [4] https://tech.feedyourhead.at/content/copy-partition-table-one-disk-another
• [5] https://tinyurl.com/yzf2kw4y
• [6] https://newrepublic.com/article/156829/happened-jordan-peterson
• [7] https://tinyurl.com/yfydxb9r
• [8] https://tinyurl.com/ydl9lajf
• [9] https://tinyurl.com/ygfhdjj9
• [10] https://www.wired.com/2012/07/people-staring-at-computers/

The Python programming language is one of the most popular and in huge demand. It is free, has a large community, is intended for the development of projects of varying complexity, is easy to learn, and opens up great opportunities for programmers. To work comfortably with it, you need special Python tools, which are able to simplify your work. We have selected the best Python tools that will be relevant in 2021.

Popular Python Tools 2021Python tools make life much easier for any developer and provide ample opportunities for creating effective applications or sites. These solutions help to automate different processes and minimize routine tasks.In fact, their functionality varies considerably. Some are made for full-fledged complex multi-level development, while others have a simplified interface that allows you to develop individual modules and blocks. Before choosing a tool, you need to define objectives and understand goals. In this case, it will become clear what exactly to use.

MailtrapAs you may probably know, in order to send an email, you need SMTP (Simple Mail Transfer Protocol). This is because you can't just send a letter to the recipient. It needs to be sent to the server from which the recipient will download this letter using IMAP and POP3.Mailtrap provides an opportunity to send emails in python. Moreover, Mailtrap provides #rest #api to access current emails. It can be used to automate email testing, which will improve your email marketing campaigns. For example, you can check the password recovery form in the Selenium Test and immediately see if an email was sent to the correct address. Then take a new password from the email and try to enter the site with it. Cool, isn't it?

Pros

• All emails are in one place.
• Mailtrap provides multiple inboxes.
• Shared access is present.
• It is easy to set up.
• RESTful API

ConsNo visible disadvantages were found.

Django

Django is a free and open-source full-stack framework. It is one of the most important and popular among Python developers. It helps you move from a prototype to a ready-made working solution in a short time since its main task is to automate processes and speed up work through associations and libraries. It s a great choice for a product launch.You can use Django if at least a few of the following points interest you:

• There is a need to develop the server-side of the API.
• You need to develop a web application.
• In the course of work, many changes are made, you have to constantly deploy the application and make edits.
• There are many complex tasks that are difficult to solve on your own, and you will need the help of the community.
• ORM support is needed to avoid accessing the database directly.
• There is a need to integrate new technologies such as machine learning.

Django is a great Python Web Framework that does its job. It is not for nothing that it is one of the most popular, and is actively used by millions of developers.

ProsDjango has quite a few advantages. It contains a large number of ready-made solutions, which greatly simplifies development. Admin panel, database migration, various forms, user authentication tools are extremely helpful. The structure is very clear and simple.A large community helps to solve almost any problem. Thanks to ORM, there is a high level of security and it is comfortable to work with databases.

ConsDespite its powerful capabilities, Django's Python Web Framework has drawbacks. It is very massive, monolithic, therefore it develops slowly. Despite the many generic modules, the development speed of Django itself is reduced.

CherryPy

CherryPy is a micro-framework. It is designed to solve specific problems, capable of running the program on any operating system. CherryPy is used in the following cases:

• To create an application with small code size.
• There is a need to manage several servers at the same time.
• You need to monitor the performance of applications.

CherryPy refers to Python Frameworks, which are designed for specific tasks. It's clear, user-friendly, and ideal for Android development.

ProsCherryPy Python tool has a friendly and understandable development environment. This is a functional and complete framework, which can be used to build good applications. The source code is open, so the platform is completely free for developers, and the community, although not too large, is very responsive, and always helps to solve problems.

ConsThere are not so many cons to this Python tool. It is not capable of performing complex tasks and functions, it is intended more for specific solutions, for example, for the development of certain plugins or modules.

Pyramid

Python Pyramid tool is designed for programming complex objects and solving multifunctional problems. It is used by professional programmers and is traditionally used for identification and routing. It is aimed at a wide audience and is capable of developing API prototypes.It is used in the following cases:

• You need problem indicator tools to make timely adjustments and edits.
• You use several programming languages at once;
• You work with reporting and financial calculations, forecasting;
• You need to quickly create a simple application.

At the same time, the Python Web Framework Pyramid allows you to create complex applications with great functionality like a translation software.

ProsPyramid does an excellent job of developing basic applications quickly. It is quite flexible and easy to learn. In fact, the key to the success of this framework is that it is completely based on fundamental principles, using simple and basic programming techniques. It is minimalistic, but at the same time offers users a lot of freedom of action. It is able to work with both small applications and powerful multifunctional programs.

ConsIt is difficult to deviate from the basic principles. This Python tool makes the decision for you. Simple programs are very easy to implement. But to do something complex and large-scale, you have to completely immerse yourself in the study of the environment and obey it.

Grok

Grok is a Python tool that works with templates. Its main task is to eliminate repetitions in the code. If the element is repeated, then the template that was already created earlier is simply applied. This greatly simplifies and speeds up the work.Grok suits developers in the following cases:

• If a programmer has little experience and is not yet ready to develop his modules.
• There is a need to quickly develop a simple application.
• The functionality of the application is simple, straightforward, and the interface does not play a key role.

ProsThe Grok framework is a child of Zope3, which was released earlier. It has a simplified structure of work, easy installation of modules, more capabilities, and better flexibility. It is designed to develop small applications. Yes, it is not intended for complex work, but due to its functionality, it allows you to quickly implement a project.

ConsThe Grok community is not very large, as this Python tool has not gained widespread popularity. Nevertheless, it is used by Python adepts for comfortable development. It is impossible to implement complex tasks on it since the possibilities are quite limited.Grok is one of the best Python Web Frameworks. It is understandable and has enough features for comfortable development.

Web2Py

Web2Py is a Python tool that has its own IDEwhich, which includes a code editor, debugger, and deployment. It works great without the need for configuration or installation, provides a high level of data security, and is suitable for work on various platforms.Web2Py is great in the following cases:

• When there is a need to develop something on different operating systems.
• If there is no way to install and configure the framework.
• When a high level of data security is required, for example, when developing financial applications or sales performance management tools.
• If you need to carefully track bugs right during development, and not during the testing phase.

ProsWeb2Py is capable of working with different protocols, has a built-in error tracker, and has a backward compatibility feature that helps to work on the basis of previous versions of the framework. This means that code maintenance becomes much easier and cheaper. It's free, open-source, and very flexible.

ConsAmong the many Python tools, there are not many that require the latest version of the language. Web2Py is one of those and won't work on Python 3 and below. Therefore, you need to constantly monitor the updates.Web2Py does an excellent job of its tasks. It is quite simple and accessible to everyone.

BlueBream

BlueBream used to be called Zope3 before. It copes well with tasks of the medium and high level of complexity and is suitable for working on serious projects.

ProsThe BlueBream build system is quite powerful and suitable for complex tasks. You can create functional applications on it, and the principle of reuse of components makes the code easier. At the same time, the speed of development increases. The software can be scaled, and a transactional object database provides an easy path to store it. This means that queries are processed quickly and database management is simple.

ConsThis is not a very flexible framework, it is better to know clearly in advance what is required of it. In addition, it cannot withstand heavy loads. When working with 1000 users at the same time, it can crash and give errors. Therefore, it should be used to solve narrow problems.Python frameworks are often designed for specific tasks. BlueBream is one of these and is suitable for applications where database management plays a key role.

ConclusionPython tools come in different forms and have vastly different capabilities. There are quite a few of them, but in 2021 these will be the most popular and in demand. Experienced programmers always choose several development tools for their comfortable work.

LTS This is my third month of working for LTS. I was assigned 12 hrs and worked all of them.

Released DLAs

1. DLA-2646-1 subversion_1.9.5-1+deb9u6
   • CVE-2020-17525: Remote unauthenticated denial-of-service in Subversion mod_authz_svn
2. DLA-2649-1 cgal_4.9-1+deb9u1
   • CVE-2020-28601: An oob read vulnerability exists in Nef_2/PM_io_parser.h PM_io_parser::read_vertex() Face_of[] OOB read. An attacker can provide malicious input to trigger this vulnerability.
   • CVE-2020-28636: An oob read vulnerability exists in Nef_S2/SNC_io_parser.h SNC_io_parser::read_sloop() slh->twin() An attacker can provide malicious input to trigger this vulnerability.
   • CVE-2020-35628: An oob read vulnerability exists in Nef_S2/SNC_io_parser.h SNC_io_parser::read_sloop() slh->incident_sface. An attacker can provide malicious input to trigger this vulnerability.
   • CVE-2020-35636: An oob read vulnerability exists in Nef_S2/SNC_io_parser.h SNC_io_parser::read_sface() sfh->volume(). An attacker can provide malicious input to trigger this vulnerability.
3. DLA-2660-1 libgetdata_0.9.4-1+deb9u1
   • CVE-2021-20204: A heap memory corruption problem (use after free) can be triggered when processing maliciously crafted dirfile databases. This degrades the confidentiality, integrity and availability of third-party software that uses libgetdata as a library.

Other LTS-related work

bind9 LTS-repo on salsa for testing Created a repo for bind9 to test the package in he salsa-pipeline. Package testing was asked in the mailing list. After that I have added autopkgtests, which were copied from the main salsa-repo and updated to stretch release.

libwebp and imagemagick Two packages with a high number of CVEs were in my focus this month. The work is not yet finished and DLAs will be released soon.

Debian Science Team I have prepared and uploaded following packages, which are maintained under the umbrella of Debian Science Team:

• gfsview_20121130+dfsg-7, fixed RC-Bug #987935 and created ci-pipeline for the package (team upload). And requested the package unblock #988112.
• Reviewed and sponsored linbox_1.6.3-3 (RC-Bug #987921)
• Prepared and uploaded libgetdata_0.10.0-5+deb10u1, fixing CVE-2021-20204 in buster (through proposed-updates)
• Reviewed and sponsored freefem++_3.61.1+dfsg1-6 (RC-Bug #957233)
• Prepared and uploaded sundials_4.1.0+dfsg-4 (RC-Bug #988551)

I previously built a single node Kubernetes cluster as a test environment to learn more about it. The first thing I want to try to understand is its networking. In particular the IP addresses that are listed are all 10.* and my host s network is a 192.168/24. I understand each pod gets its own virtual ethernet interface and associated IP address, and these are generally private within the cluster (and firewalled out other than for exposed services). What does that actually look like? Huh. No sign of any way to get to 10.107.66.138 (the IP my echoserver from the previous post is available on directly from the host). What about network interfaces? (under the cut because it s lengthy)

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enx00e04c6851de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:4c:68:51:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.53.147/24 brd 192.168.53.255 scope global dynamic enx00e04c6851de
       valid_lft 41571sec preferred_lft 41571sec
3: wlp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 74:d8:3e:70:3b:18 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:18:04:9e:08 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d2:5a:fd:c1:56:23 brd ff:ff:ff:ff:ff:ff
7: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
    link/ether 12:82:8f:ed:c7:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global weave
       valid_lft forever preferred_lft forever
9: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master datapath state UP group default
    link/ether b6:49:88:d6:6d:84 brd ff:ff:ff:ff:ff:ff
10: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether 6e:6c:03:1d:e5:0e brd ff:ff:ff:ff:ff:ff
11: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master datapath state UNKNOWN group default qlen 1000
    link/ether 9a:af:c5:0a:b3:fd brd ff:ff:ff:ff:ff:ff
13: vethwepl534c0a6@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether 1e:ac:f1:85:61:9a brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: vethwepl9ffd6b6@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether 56:ca:71:2a:ab:39 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: vethwepl62b369d@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether e2:a0:bb:ee:fc:73 brd ff:ff:ff:ff:ff:ff link-netnsid 2
23: vethwepl6669168@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether f2:e7:e6:95:e0:61 brd ff:ff:ff:ff:ff:ff link-netnsid 3

That looks like a collection of virtual ethernet devices that are being managed by the weave networking plugin, and presumably partnered inside each pod. They re bridged to the weave interface (the master weave bit). Still no clues about the 10.* range. What about ARP?

ip neigh

192.168.53.1 dev enx00e04c6851de lladdr e4:8d:8c:35:98:d5 DELAY
192.168.0.4 dev datapath lladdr da:22:06:96:50:cb STALE
192.168.0.2 dev weave lladdr 66:eb:ce:16:3c:62 REACHABLE
192.168.53.136 dev enx00e04c6851de lladdr 00:e0:4c:39:f2:54 REACHABLE
192.168.0.6 dev weave lladdr 56:a9:f0:d2:9e:f3 STALE
192.168.0.3 dev datapath lladdr f2:42:c9:c3:08:71 STALE
192.168.0.3 dev weave lladdr f2:42:c9:c3:08:71 REACHABLE
192.168.0.2 dev datapath lladdr 66:eb:ce:16:3c:62 STALE
192.168.0.6 dev datapath lladdr 56:a9:f0:d2:9e:f3 STALE
192.168.0.4 dev weave lladdr da:22:06:96:50:cb STALE
192.168.0.5 dev datapath lladdr fe:6f:1b:14:56:5a STALE
192.168.0.5 dev weave lladdr fe:6f:1b:14:56:5a REACHABLE

Nope. That just looks like addresses on the weave managed bridge. Alright. What about firewalling?

nft list ruleset

table ip nat  
	chain DOCKER  
		iifname "docker0" counter packets 0 bytes 0 return
	 
	chain POSTROUTING  
		type nat hook postrouting priority srcnat; policy accept;
		 counter packets 531750 bytes 31913539 jump KUBE-POSTROUTING
		oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 1 bytes 84 masquerade 
		counter packets 525600 bytes 31544134 jump WEAVE
	 
	chain PREROUTING  
		type nat hook prerouting priority dstnat; policy accept;
		 counter packets 180 bytes 12525 jump KUBE-SERVICES
		fib daddr type local counter packets 23 bytes 1380 jump DOCKER
	 
	chain OUTPUT  
		type nat hook output priority -100; policy accept;
		 counter packets 527005 bytes 31628455 jump KUBE-SERVICES
		ip daddr != 127.0.0.0/8 fib daddr type local counter packets 285425 bytes 17125524 jump DOCKER
	 
	chain KUBE-MARK-DROP  
		counter packets 0 bytes 0 meta mark set mark or 0x8000 
	 
	chain KUBE-MARK-MASQ  
		counter packets 0 bytes 0 meta mark set mark or 0x4000 
	 
	chain KUBE-POSTROUTING  
		mark and 0x4000 != 0x4000 counter packets 4622 bytes 277720 return
		counter packets 0 bytes 0 meta mark set mark xor 0x4000 
		 counter packets 0 bytes 0 masquerade 
	 
	chain KUBE-KUBELET-CANARY  
	 
	chain INPUT  
		type nat hook input priority 100; policy accept;
	 
	chain KUBE-PROXY-CANARY  
	 
	chain KUBE-SERVICES  
		meta l4proto tcp ip daddr 10.96.0.10  tcp dport 9153 counter packets 0 bytes 0 jump KUBE-SVC-JD5MR3NA4I4DYORP
		meta l4proto tcp ip daddr 10.107.66.138  tcp dport 8080 counter packets 1 bytes 60 jump KUBE-SVC-666FUMINWJLRRQPD
		meta l4proto tcp ip daddr 10.111.16.129  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-EZYNCFY2F7N6OQA2
		meta l4proto tcp ip daddr 10.96.9.41  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-EDNDUDH2C75GIR6O
		meta l4proto tcp ip daddr 192.168.53.147  tcp dport 443 counter packets 0 bytes 0 jump KUBE-XLB-EDNDUDH2C75GIR6O
		meta l4proto tcp ip daddr 10.96.9.41  tcp dport 80 counter packets 0 bytes 0 jump KUBE-SVC-CG5I4G2RS3ZVWGLK
		meta l4proto tcp ip daddr 192.168.53.147  tcp dport 80 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
		meta l4proto tcp ip daddr 10.96.0.1  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-NPX46M4PTMTKRN6Y
		meta l4proto udp ip daddr 10.96.0.10  udp dport 53 counter packets 0 bytes 0 jump KUBE-SVC-TCOU7JCQXEZGVUNU
		meta l4proto tcp ip daddr 10.96.0.10  tcp dport 53 counter packets 0 bytes 0 jump KUBE-SVC-ERIFXISQEP7F7OF4
		 fib daddr type local counter packets 3312 bytes 198720 jump KUBE-NODEPORTS
	 
	chain KUBE-NODEPORTS  
		meta l4proto tcp  tcp dport 31529 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp  tcp dport 31529 counter packets 0 bytes 0 jump KUBE-SVC-666FUMINWJLRRQPD
		meta l4proto tcp ip saddr 127.0.0.0/8  tcp dport 30894 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp  tcp dport 30894 counter packets 0 bytes 0 jump KUBE-XLB-EDNDUDH2C75GIR6O
		meta l4proto tcp ip saddr 127.0.0.0/8  tcp dport 32740 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp  tcp dport 32740 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
	 
	chain KUBE-SVC-NPX46M4PTMTKRN6Y  
		 counter packets 0 bytes 0 jump KUBE-SEP-Y6PHKONXBG3JINP2
	 
	chain KUBE-SEP-Y6PHKONXBG3JINP2  
		ip saddr 192.168.53.147  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.53.147:6443
	 
	chain WEAVE  
		# match-set weaver-no-masq-local dst  counter packets 135966 bytes 8160820 return
		ip saddr 192.168.0.0/24 ip daddr 224.0.0.0/4 counter packets 0 bytes 0 return
		ip saddr != 192.168.0.0/24 ip daddr 192.168.0.0/24 counter packets 0 bytes 0 masquerade 
		ip saddr 192.168.0.0/24 ip daddr != 192.168.0.0/24 counter packets 33 bytes 2941 masquerade 
	 
	chain WEAVE-CANARY  
	 
	chain KUBE-SVC-JD5MR3NA4I4DYORP  
		  counter packets 0 bytes 0 jump KUBE-SEP-6JI23ZDEH4VLR5EN
		 counter packets 0 bytes 0 jump KUBE-SEP-FATPLMAF37ZNQP5P
	 
	chain KUBE-SEP-6JI23ZDEH4VLR5EN  
		ip saddr 192.168.0.2  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.2:9153
	 
	chain KUBE-SVC-TCOU7JCQXEZGVUNU  
		  counter packets 0 bytes 0 jump KUBE-SEP-JTN4UBVS7OG5RONX
		 counter packets 0 bytes 0 jump KUBE-SEP-4TCKAEJ6POVEFPVW
	 
	chain KUBE-SEP-JTN4UBVS7OG5RONX  
		ip saddr 192.168.0.2  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto udp   counter packets 0 bytes 0 dnat to 192.168.0.2:53
	 
	chain KUBE-SVC-ERIFXISQEP7F7OF4  
		  counter packets 0 bytes 0 jump KUBE-SEP-UPZX2EM3TRFH2ASL
		 counter packets 0 bytes 0 jump KUBE-SEP-KPHYKKPVMB473Z76
	 
	chain KUBE-SEP-UPZX2EM3TRFH2ASL  
		ip saddr 192.168.0.2  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.2:53
	 
	chain KUBE-SEP-4TCKAEJ6POVEFPVW  
		ip saddr 192.168.0.3  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto udp   counter packets 0 bytes 0 dnat to 192.168.0.3:53
	 
	chain KUBE-SEP-KPHYKKPVMB473Z76  
		ip saddr 192.168.0.3  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.3:53
	 
	chain KUBE-SEP-FATPLMAF37ZNQP5P  
		ip saddr 192.168.0.3  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.3:9153
	 
	chain KUBE-SVC-666FUMINWJLRRQPD  
		 counter packets 1 bytes 60 jump KUBE-SEP-LYLDBZYLHY4MT3AQ
	 
	chain KUBE-SEP-LYLDBZYLHY4MT3AQ  
		ip saddr 192.168.0.4  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 1 bytes 60 dnat to 192.168.0.4:8080
	 
	chain KUBE-XLB-EDNDUDH2C75GIR6O  
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-EDNDUDH2C75GIR6O
		 counter packets 0 bytes 0 jump KUBE-SEP-BLQHCYCSXY3NRKLC
	 
	chain KUBE-XLB-CG5I4G2RS3ZVWGLK  
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-CG5I4G2RS3ZVWGLK
		 counter packets 0 bytes 0 jump KUBE-SEP-5XVRKWM672JGTWXH
	 
	chain KUBE-SVC-EDNDUDH2C75GIR6O  
		 counter packets 0 bytes 0 jump KUBE-SEP-BLQHCYCSXY3NRKLC
	 
	chain KUBE-SEP-BLQHCYCSXY3NRKLC  
		ip saddr 192.168.0.5  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.5:443
	 
	chain KUBE-SVC-CG5I4G2RS3ZVWGLK  
		 counter packets 0 bytes 0 jump KUBE-SEP-5XVRKWM672JGTWXH
	 
	chain KUBE-SEP-5XVRKWM672JGTWXH  
		ip saddr 192.168.0.5  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.5:80
	 
	chain KUBE-SVC-EZYNCFY2F7N6OQA2  
		 counter packets 0 bytes 0 jump KUBE-SEP-JYW326XAJ4KK7QPG
	 
	chain KUBE-SEP-JYW326XAJ4KK7QPG  
		ip saddr 192.168.0.5  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.5:8443
	 
 
table ip filter  
	chain DOCKER  
	 
	chain DOCKER-ISOLATION-STAGE-1  
		iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
		counter packets 0 bytes 0 return
	 
	chain DOCKER-ISOLATION-STAGE-2  
		oifname "docker0" counter packets 0 bytes 0 drop
		counter packets 0 bytes 0 return
	 
	chain FORWARD  
		type filter hook forward priority filter; policy drop;
		iifname "weave"  counter packets 213 bytes 54014 jump WEAVE-NPC-EGRESS
		oifname "weave"  counter packets 150 bytes 30038 jump WEAVE-NPC
		oifname "weave" ct state new counter packets 0 bytes 0 log group 86 
		oifname "weave" counter packets 0 bytes 0 drop
		iifname "weave" oifname != "weave" counter packets 33 bytes 2941 accept
		oifname "weave" ct state related,established counter packets 0 bytes 0 accept
		 counter packets 0 bytes 0 jump KUBE-FORWARD
		ct state new  counter packets 0 bytes 0 jump KUBE-SERVICES
		ct state new  counter packets 0 bytes 0 jump KUBE-EXTERNAL-SERVICES
		counter packets 0 bytes 0 jump DOCKER-USER
		counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
		oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
		oifname "docker0" counter packets 0 bytes 0 jump DOCKER
		iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
		iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
	 
	chain DOCKER-USER  
		counter packets 0 bytes 0 return
	 
	chain KUBE-FIREWALL  
		 mark and 0x8000 == 0x8000 counter packets 0 bytes 0 drop
		ip saddr != 127.0.0.0/8 ip daddr 127.0.0.0/8  ct status dnat counter packets 0 bytes 0 drop
	 
	chain OUTPUT  
		type filter hook output priority filter; policy accept;
		ct state new  counter packets 527014 bytes 31628984 jump KUBE-SERVICES
		counter packets 36324809 bytes 6021214027 jump KUBE-FIREWALL
		meta l4proto != esp  mark and 0x20000 == 0x20000 counter packets 0 bytes 0 drop
	 
	chain INPUT  
		type filter hook input priority filter; policy accept;
		 counter packets 35869492 bytes 5971008896 jump KUBE-NODEPORTS
		ct state new  counter packets 390938 bytes 23457377 jump KUBE-EXTERNAL-SERVICES
		counter packets 36249774 bytes 6030068622 jump KUBE-FIREWALL
		meta l4proto tcp ip daddr 127.0.0.1 tcp dport 6784 fib saddr type != local ct state != related,established  counter packets 0 bytes 0 drop
		iifname "weave" counter packets 907273 bytes 88697229 jump WEAVE-NPC-EGRESS
		counter packets 34809601 bytes 5818213726 jump WEAVE-IPSEC-IN
	 
	chain KUBE-KUBELET-CANARY  
	 
	chain KUBE-PROXY-CANARY  
	 
	chain KUBE-EXTERNAL-SERVICES  
	 
	chain KUBE-NODEPORTS  
		meta l4proto tcp  tcp dport 32196 counter packets 0 bytes 0 accept
		meta l4proto tcp  tcp dport 32196 counter packets 0 bytes 0 accept
	 
	chain KUBE-SERVICES  
	 
	chain KUBE-FORWARD  
		ct state invalid counter packets 0 bytes 0 drop
		 mark and 0x4000 == 0x4000 counter packets 0 bytes 0 accept
		 ct state related,established counter packets 0 bytes 0 accept
		 ct state related,established counter packets 0 bytes 0 accept
	 
	chain WEAVE-NPC-INGRESS  
	 
	chain WEAVE-NPC-DEFAULT  
		# match-set weave-;rGqyMIl1HN^cfDki~Z$3]6!N dst  counter packets 14 bytes 840 accept
		# match-set weave-P.B !ZhkAr5q=XZ?3 tMBA+0 dst  counter packets 0 bytes 0 accept
		# match-set weave-Rzff h:=]JaaJl/G;(XJpGjZ[ dst  counter packets 0 bytes 0 accept
		# match-set weave-]B*(W?)t*z5O17G044[gUo#$l dst  counter packets 0 bytes 0 accept
		# match-set weave-iLgO^ o=U/*%KE[@=W:l~ 9T dst  counter packets 9 bytes 540 accept
	 
	chain WEAVE-NPC  
		ct state related,established counter packets 124 bytes 28478 accept
		ip daddr 224.0.0.0/4 counter packets 0 bytes 0 accept
		# PHYSDEV match --physdev-out vethwe-bridge --physdev-is-bridged counter packets 3 bytes 180 accept
		ct state new counter packets 23 bytes 1380 jump WEAVE-NPC-DEFAULT
		ct state new counter packets 0 bytes 0 jump WEAVE-NPC-INGRESS
	 
	chain WEAVE-NPC-EGRESS-ACCEPT  
		counter packets 48 bytes 3769 meta mark set mark or 0x40000 
	 
	chain WEAVE-NPC-EGRESS-CUSTOM  
	 
	chain WEAVE-NPC-EGRESS-DEFAULT  
		# match-set weave-s_+ChJId4Uy_$ G;WdH ~TK)I src  counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-s_+ChJId4Uy_$ G;WdH ~TK)I src  counter packets 0 bytes 0 return
		# match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src  counter packets 31 bytes 2749 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src  counter packets 31 bytes 2749 return
		# match-set weave-41s)5vQ^o/xWGz6a20N:~?# E src  counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-41s)5vQ^o/xWGz6a20N:~?# E src  counter packets 0 bytes 0 return
		# match-set weave-sui%__gZ kX~oZgI_Ttqp=Dp src  counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-sui%__gZ kX~oZgI_Ttqp=Dp src  counter packets 0 bytes 0 return
		# match-set weave-nmMUaDKV*YkQcP5s?Q[R54Ep3 src  counter packets 17 bytes 1020 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-nmMUaDKV*YkQcP5s?Q[R54Ep3 src  counter packets 17 bytes 1020 return
	 
	chain WEAVE-NPC-EGRESS  
		ct state related,established counter packets 907425 bytes 88746642 accept
		# PHYSDEV match --physdev-in vethwe-bridge --physdev-is-bridged counter packets 0 bytes 0 return
		fib daddr type local counter packets 11 bytes 640 return
		ip daddr 224.0.0.0/4 counter packets 0 bytes 0 return
		ct state new counter packets 50 bytes 3961 jump WEAVE-NPC-EGRESS-DEFAULT
		ct state new mark and 0x40000 != 0x40000 counter packets 2 bytes 192 jump WEAVE-NPC-EGRESS-CUSTOM
	 
	chain WEAVE-IPSEC-IN  
	 
	chain WEAVE-CANARY  
	 
 
table ip mangle  
	chain KUBE-KUBELET-CANARY  
	 
	chain PREROUTING  
		type filter hook prerouting priority mangle; policy accept;
	 
	chain INPUT  
		type filter hook input priority mangle; policy accept;
		counter packets 35716863 bytes 5906910315 jump WEAVE-IPSEC-IN
	 
	chain FORWARD  
		type filter hook forward priority mangle; policy accept;
	 
	chain OUTPUT  
		type route hook output priority mangle; policy accept;
		counter packets 35804064 bytes 5938944956 jump WEAVE-IPSEC-OUT
	 
	chain POSTROUTING  
		type filter hook postrouting priority mangle; policy accept;
	 
	chain KUBE-PROXY-CANARY  
	 
	chain WEAVE-IPSEC-IN  
	 
	chain WEAVE-IPSEC-IN-MARK  
		counter packets 0 bytes 0 meta mark set mark or 0x20000
	 
	chain WEAVE-IPSEC-OUT  
	 
	chain WEAVE-IPSEC-OUT-MARK  
		counter packets 0 bytes 0 meta mark set mark or 0x20000
	 
	chain WEAVE-CANARY  
	 
 

Wow. That s a lot of nftables entries, but it explains what s going on. We have a nat entry for: which ends up going to KUBE-SEP-LYLDBZYLHY4MT3AQ and: So packets headed for our echoserver are eventually ending up in a container that has a local IP address of 192.168.0.4. Which we can see in our routing table via the weave interface. Mystery explained. We can see the ingress for the externally visible HTTP service as well: which ends up redirected to: So from that we d expect the IP inside the echoserver pod to be 192.168.0.4 and the IP address instead our nginx ingress pod to be 192.168.0.5. Let s look: It s a slightly awkward method of determining the local IPs addresses due to the stripped down nature of the container, but it clearly shows the expected 192.168.0.4 address. I ve touched here upon the ability to actually enter a container and have a poke around its running environment by using docker directly. Next step is to use that to investigate what containers have actually been spun up and what they re doing. I ll also revisit networking when I get to the point of building a multi-node cluster, to examine how the bridging between different hosts is done.

As it is now known, the freenode IRC network has been taken over by a Trumpian wannabe korean royalty bitcoins millionaire. To make a long story short, the former freenode head of staff secretly "sold" the network to this person even if it was not hers to sell, and our lawyers have advised us that there is not much that we can do about it without some of us risking financial ruin. Fuck you Christel, lilo's life work did not deserve this. What you knew as freenode after 12:00 UTC of May 19 will be managed by different people. As I have no desire to volunteer under the new regime, this marks the end of my involvement with freenode. It had started in 1999 when I encouraged the good parts of #linux-it to leave ircnet, and soon after I became senior staff. Even if I have not been very active recently, at this point I was the longest-serving freenode staff member and now I expect that I will hold this record forever. The people that I have met on IRC, on freenode and other networks, have been and still are a very important part of my life, second only to the ones that I have known thanks to Usenet. I am not fine, but I know that the communities which I have been a part of are not defined by a domain name and will regroup somewhere else. The current freenode staff members have resigned with me, these are some of their farewell messages:

amdj

edk

emilsp

Fuchs

jess

JonathanD

kline

niko

mniip

Swant

Together we have created Libera.Chat, a new IRC network based on the same principles of the old freenode.

The Population Issue and its many facets Another couple of weeks passed. A Lot of things happening, lots of anger and depression in folks due to handling in pandemic, but instead of blaming they are willing to blame everybody else including the population. Many of them want forced sterilization like what Sanjay Gandhi did during the Emergency (1975). I had to share So Long, My son . A very moving tale of two families of what happened to them during the one-child policy in China. I was so moved by it and couldn t believe that the Chinese censors allowed it to be produced, shot, edited, and then shared worldwide. It also won a couple of awards at the 69th Berlin Film Festival, silver bear for the best actor and the actress in that category. But more than the award, the theme, and the concept as well as the length of the movie which was astonishing. Over a 3 hr. something it paints a moving picture of love, loss, shame, relief, anger, and asking for forgiveness. All of which can be identified by any rational person with feelings worldwide.

Girl child What was also interesting though was what it couldn t or wasn t able to talk about and that is the Chinese leftover men. In fact, a similar situation exists here in India, only it has been suppressed. This has been more pronounced more in Asia than in other places. One big thing in this is human trafficking and mostly women trafficking. For the Chinese male, that was happening on a large scale from all neighboring countries including India. This has been shared in media and everybody knows about it and yet people are silent. But this is not limited to just the Chinese, even Indians have been doing it. Even yesteryear actress Rupa Ganguly was caught red-handed but then later let off after formal questioning as she is from the ruling party. So much for justice. What is and has been surprising at least for me is Rwanda which is in the top 10 of some of the best places in equal gender. It, along with other African countries have also been in news for putting quite a significant amount of percentage of GDP into public healthcare (between 20-10%), but that is a story for a bit later. People forget or want to forget that it was in Satara, a city in my own state where 220 girls changed their name from nakusha or unwanted to something else and that had become a piece of global news. One would think that after so many years, things would have changed, the only change that has happened is that now we have two ministries, The Ministry of Women and Child Development (MoWCD) and The Ministry of Health and Welfare (MoHFW). Sadly, in both cases, the ministries have been found wanting, Whether it was the high-profile Hathras case or even the routine cries of help which given by women on the twitter helpline. Sadly, neither of these ministries talks about POSH guidelines which came up after the 2012 gangrape case. For both these ministries, it should have been a pinned tweet. There is also the 1994 PCPNDT Act which although made in 1994, actually functioned in 2006, although what happens underground even today nobody knows . On the global stage, about a decade ago, Stephen J. Dubner and Steven Levitt argued in their book Freakonomics how legalized abortion both made the coming population explosion as well as expected crime rates to be reduced. There was a huge pushback on the same from the conservatives and has become a matter of debate, perhaps something that the Conservatives wanted. Interestingly, it hasn t made them go back but go forward as can be seen from the Freakonomics site.

Climate Change Another topic that came up for discussion was repeatedly climate change, but when I share Shell s own 1998 Confidential report titled Greenhouse effect all become strangely silent. The silence here is of two parts, there probably is a large swathe of Indians who haven t read the report and there may be a minority who have read it and know what already has been shared with U.S. Congress. The Conservative s argument has been for it is jobs and a weak we need to research more . There was a partial debunk of it on the TBD podcast by Matt Farell and his brother Sean Farell as to how quickly the energy companies are taking to the coming change.

Health Budget Before going to Covid stories. I first wanted to talk about Health Budgets. From the last 7 years the Center s allocation for health has been between 0.34 to 0.8% per year. That amount barely covers the salaries to the staff, let alone any money for equipment or anything else. And here by allocation I mean, what is actually spent, not the one that is shared by GOI as part of budget proposal. In fact, an article on Wire gives a good breakdown of the numbers. Even those who are on the path of free markets describe India s health business model as a flawed one. See the Bloomberg Quint story on that. Now let me come to Rwanda. Why did I chose Rwanda, I could have chosen South Africa where I went for Debconf 2016, I chose because Rwanda s story is that much more inspiring. In many ways much more inspiring than that South Africa in many ways. Here is a country which for decades had one war or the other, culminating into the Rwanda Civil War which ended in 1994. And coincidentally, they gained independence on a similar timeline as South Africa ending Apartheid in 1994. What does the country do, when it gains its independence, it first puts most of its resources in the healthcare sector. The first few years at 20% of GDP, later than at 10% of GDP till everybody has universal medical coverage. Coming back to the Bloomberg article I shared, the story does not go into the depth of beyond-expiry date medicines, spurious medicines and whatnot. Sadly, most media in India does not cover the deaths happening in rural areas and this I am talking about normal times. Today what is happening in rural areas is just pure madness. For last couple of days have been talking with people who are and have been covering rural areas. In many of those communities, there is vaccine hesitancy and why, because there have been whatsapp forwards sharing that if you go to a hospital you will die and your kidney or some other part of the body will be taken by the doctor. This does two things, it scares people into not going and getting vaccinated, at the same time they are prejudiced against science. This is politics of the lowest kind. And they do it so that they will be forced to go to temples or babas and what not and ask for solutions. And whether they work or not is immaterial, they get fixed and property and money is seized. Sadly, there are not many Indian movies of North which have tried to show it except for oh my god but even here it doesn t go the distance. A much more honest approach was done in Trance . I have never understood how the South Indian movies are able to do a more honest job of story-telling than what is done in Bollywood even though they do in 1/10th the budget that is needed in Bollywood. Although, have to say with OTT, some baggage has been shed but with the whole film certification rearing its ugly head through MEITY orders, it seems two steps backward instead of forward. The idea being simply to infantilize the citizens even more. That is a whole different ball-game which probably will require its own space.

Vaccine issues One good news though is that Vaccination has started. But it has been a long story full of greed by none other than GOI (Government of India) or the ruling party BJP. Where should I start with. I probably should start with this excellent article done by Priyanka Pulla. It is interesting and fascinating to know how vaccines are made, at least one way which she shared. She also shared about the Cutter Incident which happened in the late 50 s. The response was on expected lines, character assassination of her and the newspaper they published but could not critique any of the points made by her. Not a single point that she didn t think about x or y. Interestingly enough, in January 2021 Bharati Biotech was supposed to be share phase 3 trial data but hasn t been put up in public domain till May 2021. In fact, there have been a few threads raised by both well-meaning Indians as well as others globally especially on twitter to which GOI/ICMR (Indian Council of Medical Research) is silent. Another interesting point to note is that Russia did say in its press release that it is possible that their vaccine may not be standard (read inactivation on their vaccines and another way is possible but would take time, again Brazil has objected, but India hasn t till date.) What also has been interesting is the homegrown B.1.617 lineage or known as double mutant . This was first discovered from my own state, Maharashtra and then transported around the world. There is also B.1.618 which was found in West Bengal and is same or supposed to be similar to the one found in South Africa. This one is known as Triple mutant . About B.1.618 we don t know much other than knowing that it is much more easily transferable, much more infectious. Most countries have banned flights from India and I cannot fault them anyway. Hell, when even our diplomats do not care for procedures to be followed during the pandemic then how a common man is supposed to do. Of course, now for next month, Mr. Modi was supposed to go and now will not attend the G7 meeting. Whether, it is because he would have to face the press (the only Prime Minister and the only Indian Prime Minister who never has faced free press.) or because the Indian delegation has been disinvited, we would never know.

A good article which shares lots of lows with how things have been done in India has been an article by Arundhati Roy. And while the article in itself is excellent and shares a bit of the bitter truth but is still incomplete as so much has been happening. The problem is that the issue manifests in so many ways, it is difficult to hold on. As Arundhati shared, should we just look at figures and numbers and hold on, or should we look at individual ones, for e.g. the one shared in Outlook India. Or the one shared by Dr. Dipshika Ghosh who works in Covid ICU in some hospital

Dr. Dipika Ghosh sharing an incident in Covid Ward

Interestingly as well, while in the vaccine issue, Brazil Anvisa doesn t know what they are doing or the regulator just isn t knowledgeable etc. (statements by various people in GOI, when it comes to testing kits, the same is an approver.)

ICMR/DGCI approving internationally validated kits, Press release.

Twitter In the midst of all this, one thing that many people have forgotten and seem to have forgotten that Twitter and other tools are used by only the elite. The reason why the whole thing has become serious now than in the first phase is because the elite of India have also fallen sick and dying which was not the case so much in the first phase. The population on Twitter is estimated to be around 30-34 million and people who are everyday around 20 odd million or so, which is what 2% of the Indian population which is estimated to be around 1.34 billion. The other 98% don t even know that there is something like twitter on which you can ask help. Twitter itself is exclusionary in many ways, with both the emoticons, the language and all sorts of things. There is a small subset who does use Twitter in regional languages, but they are too small to write anything about. The main language is English which does become a hindrance to lot of people.

Censorship Censorship of Indians critical of Govt. mishandling has been non-stop. Even U.S. which usually doesn t interfere into India s internal politics was forced to make an exception. But of course, this has been on deaf ears. There is and was a good thread on Twitter by Gaurav Sabnis, a friend, fellow Puneite now settled in U.S. as a professor.

Gaurav on Trump-Biden on vaccination of their own citizens

Now just to surmise what has been happened in India and what has been happening in most of the countries around the world. Most of the countries have done centralization purchasing of the vaccine and then is distributed by the States, this is what we understand as co-operative federalism. While last year, GOI took a lot of money under the shady PM Cares fund for vaccine purchase, donations from well-meaning Indians as well as Industries and trade bodies. Then later, GOI said it would leave the states hanging and it is they who would have to buy vaccines from the manufacturers. This is again cheap politics. The idea behind it is simple, GOI knows that almost all the states are strapped for cash. This is not new news, this I have shared a couple of months back. The problem has been that for the last 6-8 months no GST meeting has taken place as shared by Punjab s Finance Minister Amarinder Singh. What will happen is that all the states will fight in-between themselves for the vaccine and most of them are now non-BJP Governments. The idea is let the states fight and somehow be on top. So, the pandemic, instead of being a public health issue has become something of on which politics has to played. The news on whatsapp by RW media is it s ok even if a million or two also die, as it is India is heavily populated. Although that argument vanishes for those who lose their dear and near ones. But that just isn t the issue, the issue goes much more deeper than that Oxygen:12%
Remedisivir:12%
Sanitiser:12%
Ventilator:12%
PPE:18%
Ambulances 28% Now all the products above are essential medical equipment and should be declared as essential medical equipment and should have price controls on which GST is levied. In times of pandemic, should the center be profiting on those. States want to let go and even want the center to let go so that some relief is there to the public, while at the same time make them as essential medical equipment with price controls. But GOI doesn t want to. Leaders of opposition parties wrote open letters but no effect. What is sad to me is how Ambulances are being taxed at 28%. Are they luxury items or sin goods ? This also reminds of the recent discovery shared by Mr. Pappu Yadav in Bihar. You can see the color of ambulances as shared by Mr. Yadav, and the same news being shared by India TV news showing other ambulances. Also, the weak argument being made of not having enough drivers. Ideally, you should have 2-3 people, both 9-1-1 and Chicago Fire show 2 people in ambulance but a few times they have also shown to be flipped over. European seems to have three people in ambulance, also they are also much more disciplined as drivers, at least an opinion shared by an American expat.

Pappu Yadav, President Jan Adhikar Party, Bihar May 11, 2021

What is also interesting to note is GOI plays this game of Health is State subject and health is Central subject depending on its convenience. Last year, when it invoked the Epidemic and DMA Act it was a Central subject, now when bodies are flowing down the Ganges and pyres being lit everywhere, it becomes a State subject. But when and where money is involved, it again becomes a Central subject. The States are also understanding it, but they are fighting on too many fronts.

Snippets from Karnataka High Court hearing today, 13th March 2021

One of the good things is most of the High Courts have woken up. Many of the people on the RW think that the Courts are doing Judicial activism . And while there may be an iota of truth in it, the bitter truth is that many judges or relatives or their helpers have diagnosed and some have even died due to Covid. In face of the inevitable, what can they do. They are hauling up local Governments to make sure they are accountable while at the same time making sure that they get access to medical facilities. And I as a citizen don t see any wrong in that even if they are doing it for selfish reasons. Because, even if justice is being done for selfish reasons, if it does improve medical delivery systems for the masses, it is cool. If it means that the poor and everybody else are able to get vaccinations, oxygen and whatever they need, it is cool. Of course, we are still seeing reports of patients spending in the region of INR 50k and more for each day spent in hospital. But as there are no price controls, judges cannot do anything unless they want to make an enemy of the medical lobby in the country. A good story on medicines and what happens in rural areas, see no further than Laakhon mein ek.

Allahabad High Court hauling Uttar Pradesh Govt. for lack of Oxygen is equal to genocide, May 11, 2021

The censorship is not just related to takedown requests on twitter but nowadays also any articles which are critical of the GOI s handling. I have been seeing many articles which have shared facts and have been critical of GOI being taken down. Previously, we used to see 404 errors happen 7-10 years down the line and that was reasonable. Now we see that happen, days weeks or months. India seems to be turning more into China and North Korea and become more anti-science day-by-day

Fake websites Before going into fake websites, let me start with a fake newspaper which was started by none other than the Gujarat CM Mr. Modi in 2005 .

Gujarat Satya Samachar 2005 launched by Mr. Modi.

And if this wasn t enough than on Feb 8, 2005, he had invoked Official Secrets Act

Mr. Modi invoking Official Secrets Act, Feb 8 2005 Gujarat Samachar

The headlines were In Modi s regime press freedom is in peril-Down with Modi s dictatorship. So this was a tried and tested technique. The above information was shared by Mr. Urvish Kothari, who incidentally also has his own youtube channel. Now cut to 2021, and we have a slew of fake websites being done by the same party. In fact, it seems they started this right from 2011. A good article on BBC itself tells the story. Hell, Disinfo.eu which basically combats disinformation in EU has a whole pdf chronicling how BJP has been doing it. Some of the sites it shared are

Times of New York
Manchester Times
Times of Los Angeles
Manhattan Post
Washington Herald
and many more. The idea being take any site name which sounds similar to a brand name recognized by Indians and make fool of them. Of course, those of who use whois and other such tools can easily know what is happening. Two more were added to the list yesterday, Daily Guardian and Australia Today. There are of course, many features which tell them apart from genuine websites. Most of these are on shared hosting rather than dedicated hosting, most of these are bought either from Godaddy and Bluehost. While Bluehost used to be a class act once upon a time, both the above will do anything as long as they get money. Don t care whether it s a fake website or true. Capitalism at its finest or worst depending upon how you look at it. But most of these details are lost on people who do not know web servers, at all and instead think see it is from an exotic site, a foreign site and it chooses to have same ideas as me. Those who are corrupt or see politics as a tool to win at any cost will not see it as evil. And as a gentleman Raghav shared with me, it is so easy to fool us. An example he shared which I had forgotten. Peter England which used to be an Irish brand was bought by Aditya Birla group way back in 2000. But even today, when you go for Peter England, the way the packaging is done, the way the prices are, more often than not, people believe they are buying the Irish brand. While sharing this, there is so much of Naom Chomsky which comes to my mind again and again

Caste Issues I had written about caste issues a few times on this blog. This again came to the fore as news came that a Hindu sect used forced labor from Dalit community to make a temple. This was also shared by the hill. In both, Mr. Joshi doesn t tell that if they were volunteers then why their passports have been taken forcibly, also I looked at both minimum wage prevailing in New Jersey as a state as well as wage given to those who are in the construction Industry. Even in minimum wage, they were giving $1 when the prevailing minimum wage for unskilled work is $12.00 and as Mr. Joshi shared that they are specialized artisans, then they should be paid between $23 $30 per hour. If this isn t exploitation, then I don t know what is. And this is not the first instance, the first instance was perhaps the case against Cisco which was done by John Doe. While I had been busy with other things, it seems Cisco had put up both a demurrer petition and a petition to strike which the Court stayed. This seemed to all over again a type of apartheid practice, only this time applied to caste. The good thing is that the court stayed the petition. Dr. Ambedkar s statement if Hindus migrate to other regions on earth, Indian caste would become a world problem given at Columbia University in 1916, seems to be proven right in today s time and sadly has aged well. But this is not just something which is there only in U.S. this is there in India even today, just couple of days back, a popular actress Munmun Dutta used a casteist slur and then later apologized giving the excuse that she didn t know Hindi. And this is patently false as she has been in the Bollywood industry for almost now 16-17 years. This again, was not an isolated incident. Seema Singh, a lecturer in IIT-Kharagpur abused students from SC, ST backgrounds and was later suspended. There is an SC/ST Atrocities Act but that has been diluted by this Govt. A bit on the background of Dr. Ambedkar can be found at a blog on Columbia website. As I have shared and asked before, how do we think, for what reason the Age of Englightenment or the Age of Reason happened. If I were a fat monk or a priest who was privileges, would I have let Age of Enlightenment happen. It broke religion or rather Church which was most powerful to not so powerful and that power was more distributed among all sort of thinkers, philosophers, tinkers, inventors and so on and so forth.

Situation going forward I believe things are going to be far more complex and deadly before they get better. I had to share another term called Comorbidities which fortunately or unfortunately has also become part of twitter lexicon. While I have shared what it means, it simply means when you have an existing ailment or condition and then Coronavirus attacks you. The Virus will weaken you. The Vaccine in the best case just stops the damage, but the damage already done can t be reversed. There are people who advise and people who are taking steroids but that again has its own side-effects. And this is now, when we are in summer. I am afraid for those who have recovered, what will happen to them during the Monsoons. We know that the Virus attacks most the lungs and their quality of life will be affected. Even the immune system may have issues. We also know about the inflammation. And the grant that has been given to University of Dundee also has signs of worry, both for people like me (obese) as well as those who have heart issues already. In other news, my city which has been under partial lockdown since a month, has been extended for another couple of weeks. There are rumors that the same may continue till the year-end even if it means economics goes out of the window.There is possibility that in the next few months something like 2 million odd Indians could die

The above is a conversation between Karan Thapar and an Oxford Mathematician Dr. Murad Banaji who has shared that the under-counting of cases in India is huge. Even BBC shared an article on the scope of under-counting. Of course, those on the RW call of the evidence including the deaths and obituaries in newspapers as a narrative . And when asked that when deaths used to be in the 20 s or 30 s which has jumped to 200-300 deaths and this is just the middle class and above. The poor don t have the money to get wood and that is the reason you are seeing the bodies in Ganges whether in Buxar Bihar or Gajipur, Uttar Pradesh. The sights and visuals makes for sorry reading

Pandit Ranjan Mishra son on his father s death due to unavailability of oxygen, Varanasi, Uttar Pradesh, 11th May 2021.

For those who don t know Pandit Ranjan Mishra was a renowned classical singer. More importantly, he was the first person to suggest Mr. Modi s name as a Prime Ministerial Candidate. If they couldn t fulfil his oxygen needs, then what can be expected for the normal public.

Conclusion Sadly, this time I have no humorous piece to share, I can however share a documentary which was shared on Feluda . I have shared about Feluda or Prodosh Chandra Mitter a few times on this blog. He has been the answer of James Bond from India. I have shared previously about The Golden Fortress . An amazing piece of art by Satyajit Ray. I watched that documentary two-three times. I thought, mistakenly that I am the only fool or fan of Feluda in Pune to find out that there are people who are even more than me. There were so many facets both about Feluda and master craftsman Satyajit Ray that I was unaware about. I was just simply amazed. I even shared few of the tidbits with mum as well, although now she has been truly hooked to Korean dramas. The only solace from all the surrounding madness. So, if you have nothing to do, you can look up his books, read them and then see the movies. And my first recommendation would be the Golden Fortress. The only thing I would say, do not have high hopes. The movie is beautiful. It starts slow and then picks up speed, just like a train. So, till later. Update The Mass surveillance part I could not do justice do hence removed it at the last moment. It actually needs its whole space, article. There is so much that the Govt. is doing under the guise of the pandemic that it is difficult to share it all in one article. As it is, the article is big

I have uploaded most of my Machine Learning lecture to YouTube. The slides are in English, but the audio is in German. Some very basic contents (e.g., a demo of standard k-means clustering) were left out from this advanced class, and instead only a link to recordings from an earlier class were given. In this class, I wanted to focus on the improved (accelerated) algorithms instead. These are not included here (yet). I believe there are some contents covered in this class you will find nowhere else (yet). The first unit is pretty long (I did not split it further yet). The later units are shorter recordings. ML F1: Principles in Machine Learning

• Principles in Machine Learning
• Principles in Machine Learning /2
• Occam s Razor Principle of Parsimony
• Simple Models
• Computational Learning Theory
• Probably Approximately Correct Learning (PAC Learning) /1
• Probably Approximately Correct Learning (PAC Learning) /2
• PAC Learnable Examples
• VC Dimension (Vapnik-Chervonenkis-Dimension )
• VC Dimension Example
• Error Bounds and the VC Dimension
• No Free Lunch
• No Free Lunch Theorem
• No Free Lunch Theorem Explanation
• Bias-Variance Tradeoff
• Bias-Variance Tradeoff
• Bias vs. Variance
• Bias-Variance Decomposition
• Bias-Variance Illustration
• Different Kinds of Bias
• Data Often Has Bias
• AI Can Be Sexist and Racist
• Relationships

ML F2/F3: Correlation does not Imply Causation & Multiple Testing Problem

• Correlation does not Imply Causation
• Correlation does not Imply Causation /2
• Correlation does not Imply Causation /3
• Correlation with Statistics Classes
• Multiple Testing Problem
• Bonferroni s Principle Multiple Testing Problem
• Multiple Testing Problem

ML F4: Overfitting beranpassung

• Overfitting
• Underfitting and Overfitting
• Overfitting Decision Tree
• Overfitting Due to Noise
• Overfitting Due to Insufficient Examples

ML F5: Fluch der Dimensionalit t Curse of Dimensionality

• Curse of Dimensionality
• Combinatorial Explosion
• Concentration of Distances
• Data is in the Margins
• Illustration: Shrinking Hyperspheres
• Illustration: Shrinking Hyperspheres /2
• Effect on Search in High Dimensionality
• Summary

ML F6: Intrinsische Dimensionalit t Intrinsic Dimensionality

• Intrinsic Dimensionality
• Estimating Intrinsic Dimensionality
• Angle-Based Intrinsic Dimensionality Intuition
• Angle-Based Intrinsic Dimensionality (ABID) /2
• Consequences & Solutions

ML F7: Distanzfunktionen und hnlichkeitsfunktionen

• Distance Functions
• Distances, Metrics and Similarities
• Distances, Metrics and Similarities /2
• Distance Functions
• Distance Functions /2
• Similarity Functions
• Distances for Binary Data
• Jaccard Coefficient for Sets
• Example Distances for Categorical Data
• Mahalanobis Distance
• Scaling & Normalization
• To Scale, or not to Scale?
• To Scale, or not to Scale? /2

ML L1: Einf hrung in die Klassifikation

• Classification
• Prediction Problems
• Classification: A Multi-Stage Process
• Classification Problem
• Example
• Process of Constructing a Model
• Process of Applying the Model

ML L2: Evaluation und Wahl von Klassifikatoren

• Evaluation and Selection of Classifiers
• Quick Recap: Classification
• Classifier Evaluation: Confusion Matrix
• Classifier Evaluation: Accuracy and Error-Rate
• Precision, Recall, and F-measure
• Classifier Evaluation: Multi-Class Confusion Matrix
• Training Accuracy vs. Accuarcy on New Data
• The Need for Validation
• Holdout Validation
• Cross-Validation
• Bootstrap Validation
• Considerations for Selecting a Model

ML L3: Bayes-Klassifikatoren

• Bayesian Classification
• Bayes Classification: Motivation
• Bayes Theorem: Review
• Optimal Bayes Classifier
• Na ve Bayes Classifier
• Probability Models for a Single Attribute
• Multivariate Gaussian Bayes Classification
• Na ve Bayes Classifier: Example
• Na ve Bayes Classifier: Computational Aspects
• Na ve Bayes Classifier: Comments & Discussion

ML L4: N chste-Nachbarn Klassifikation

• Nearest-Neighbor Classification
• Nearest Neighbor Classifier Motivation
• Nearest Neighbor Classifier: Foundations
• Nearest Neighbor Classifier: Example
• Nearest Neighbor Classification: Example
• Nearest Neighbor Decision Rules

ML L5: N chste Nachbarn und Kerndichtesch tzung

• Nearest-Neighbor as Density estimation
• Nearest Neighbor Classification and Density Estimation
• Predicting with Kernel Density Estimation with k=1,3,5,15
• Error Probability of Nearest Neighbors
• Nearest Neighbor Regression
• Nearest-Neighbor Classification: Comments & Discussion

ML L6: Lernen von Entscheidungsb umen

• Decision Tree Learning
• Example (Variant of a Dataset in )
• Decision Tree Example
• Decision Trees as Rule-based Systems
• Basic Notions
• Constructing a Decision Tree /1
• Visual Interpretation of Decision Trees on R
• Constructing a Decision Tree /2
• Decision Tree Classification: Example

ML L7: Splitkriterien bei Entscheidungsb umen

• Decision Tree Splitting
• Split for Categorical Attributes
• Split for Numeric Attributes
• Best Split Example
• Quality Measures for Splits
• Measure of Impurity: Gini Index
• Gini-Index: Example
• Information Gain
• Information Gain: Example
• Information Gain: Gain-Ratio
• Gain-Ratio: Beispiel
• Classification Error
• Gini, Entropy and Classification Error
• Comparing Split Selection Measures
• Splits for Numerical Attributes

ML L8: Ensembles und Meta-Learning: Random Forests und Gradient Boosting

• Ensembles and Meta-Learning
• Ensembles and Meta-Learning
• Error-Rate of Ensembles
• Random Forests
• Boosting
• Random Forest Classification: Example
• Gradient Boosting Classification: Example

ML L9: Support Vector Machinen - Motivation

• Support Vector Machine Motivation
• Support Vector Machines
• Support Vector Machines /2
• Finding the Best Separating Hyperplane
• Maximum Margin Hyperplane

ML L10: Affine Hyperebenen und Skalarprodukte Geometrie f r SVMs

• Affine Hyperplanes and Scalar Products
• Affine Hyperplanes
• Scalar Products
• Affine Hyperplanes /2

ML L11: Maximum Margin Hyperplane die breitest m gliche Stra e

• Maximum Margin Hyperplane
• A Na ve Attempt
• Support Vectors Separable Data
• Computing the Maximum Margin Hyperplane (MMH)
• Computing the Maximum Margin Hyperplane (MMH) /2
• Boundary of the Maximum Margin Hyperplane (MMH)
• Deriving the Primal SVM Optimization Problem

ML L12: Training Support Vector Machines

• Training Support Vector Machines
• Optimization Problem
• Karush-Kuhn-Tucker KKT Conditions
• Switching to the Dual Problem
• Classification with the Dual SVM
• Optimizing the i
• Optimizing SVMs
• Sequential Minimal Optimization
• Further Improvements

ML L13: Non-linear SVM and the Kernel Trick

• Non-linear SVM and the Kernel Trick
• Nonlinear SVM
• Nonlinear SVM /2
• Kernel Functions
• Soft Margin SVM Classifier
• Soft Margin SVM Classifier /2
• Soft Margin SVM Classifier /3
• Soft Margin SVM Classifier /4

ML L14: SVM Extensions and Conclusions

• SVM Extensions and Conclusions
• Separation of more than 2 Classes
• Support Vector Regression
• Support Vector Regression Optimization Problem
• Support Vector Regression Dual
• Support Vector Data Description (SVDD)
• SVDD Dual Problem
• Support Vector Clustering
• SVMs: Comments & Discussion

ML L15: Motivation of Neural Networks

• Neural Networks
• Biological Background (Simplified)
• Biological Background (Simplified) /2

ML L16: Threshold Logic Units

• Threshold Logic Units
• Threshold Logic Units (TLUs)
• Threshold Logic Units Example
• Geometric Interpretation of TLUs
• Exclusive-Or (XOR) Problem
• Exclusive-Or (XOR) Problem /2
• Exclusive-Or (XOR) Problem /3
• Universality of TLUs
• Mark I Perceptron

ML L17: General Artificial Neural Networks

• General Artificial Neural Networks
• Simplifying Threshold Logic Units
• Weight Matrices
• From TLUs to Multilayer Perceptrons
• Some Activation Functions
• Some Activation Functions /2
• Some Activation Functions /3
• Some Activation Functions /4

ML L18: Learning Neural Networks with Backpropagation

• Learning Neural Networks with Backpropagation
• Basic Gradient Descent
• Stochastic Gradient Descent
• Learning Single-Layer Perceptrons
• Backpropagation
• Training with Backpropagation

ML L19: Deep Neural Networks

• Deep Neural Networks
• Universal Approximation Theorem
• Deep vs. Wide Neural Networks
• High vs. Low Dimensionality
• (Early) Problems of Deep Learning
• Autoencoders
• Layer-wise Pre-Training of Deep Neural Networks
• Dropout Regularization
• Batch Normalization
• Choosing Activation Functions

ML L20: Convolutional Neural Networks

• Convolutional Neural Networks
• Depth vs. Convolutional Kernel Size
• Increasing the Training Data

ML L21: Recurrent Neural Networks and LSTM

• Recurrent Neural Networks
• Recurrent Neural Networks (RNNs) on Sequences
• Recurrent Neural Networks (RNN)
• Long-Short Term Memory (LSTM)
• Further Developments

ML L22: Conclusion Classification

• Conclusion
• Other Classifiers
• Problems of Classification

ML U1: Einleitung Clusteranalyse

• Cluster Analysis Introduction
• What is Clustering?
• What is Clustering? /2
• Applications of Clustering
• Basic Steps for Clustering

ML U2: Hierarchisches Clustering

• Hierarchical Agglomerative Clustering
• Distance of Clusters
• AGNES Agglomerative Nesting
• AGNES Agglomerative Nesting /2
• Extracting Clusters from a Dendrogram
• Benefits and Limitations of HAC

ML U3: Accelerating HAC mit Anderberg s Algorithmus

• Accelerating Hierarchical Clustering
• Complexity of Hierarchical Clustering
• Anderberg s Caching
• AGNES vs. Anderberg , NNChain , SLINK
• Example: Hierarchical Clustering with Anderberg

ML U4: k-Means Clustering

• K-means Clustering
• The Sum of Squares Objective
• The Standard Algorithm (Lloyd s Algorithm)
• Non-determinism & Non-optimality
• Initialization
• Initialization /2
• Complexity of k-Means Clustering

ML U5: Accelerating k-Means Clustering

• Accelerating k-Means Clustering
• k-Means++: Weighted Random Initialization
• Making k-means Faster
• Bounding the Distances Elkan and Hamerly
• Hamerly s k-means
• Example: k-Means Clustering with Hamerly s Algorithm
• Speedup with Hamerly, Elkan, and Exponion

ML U6: Limitations of k-Means Clustering

• Limitations of k-Means Clustering
• Benefits and Drawbacks of k-Means
• Choosing the Optimum k for k-Means
• Limitations of k-Means

ML U7: Extensions of k-Means Clustering

• Extensions of k-Means Clustering
• k-Means and Distances
• k-Means Minimizes Sum of Squares, not Euclidean Distance!
• k-Means Variations for Other Distances
• Spherical k-Means for Text Clustering
• Pre-processing and Post-processing

ML U8: Partitioning Around Medoids (k-Medoids)

• Partitioning Around Medoids (k-Medoids)
• k-medoids Clustering
• Partitioning Around Medoids
• Algorithm: Partitioning Around Medoids
• Algorithm: Partitioning Around Medoids /2
• Change in TD
• Finding the Best Swap Faster
• k-Medoids, k-Means style
• Example for the Inferiority of k-Means Style k-Medoids

ML U9: Gaussian Mixture Modeling (EM Clustering)

• Gaussian Mixture Modeling Introduction
• Expectation-Maximization in Clustering
• Fitting Multiple Gaussian Distributions
• Gaussian Mixture Modeling as E-M-Optimization
• Algorithm: EM Clustering
• Numerical Issues in GMM

ML U10: Gaussian Mixture Modeling Demo

• Gaussian Mixture Modeling Demo

ML U11: BIRCH and BETULA Clustering

• BIRCH and BETULA
• BIRCH Clustering
• BIRCH Clustering Features
• BIRCH Distances
• BIRCH CF-Tree
• BETULA Cluster Features
• BETULA Distance Computations
• Accelerating k-Means with BIRCH and BETULA
• Accelerating GMM with BETULA

ML U12: Motivation Density-Based Clustering (DBSCAN)

• Motivation Density-Based Clustering
• Density-based Clustering: Core Idea

ML U13: Density-reachable and density-connected (DBSCAN Clustering)

• Density-Based Clustering Fundamentals
• Density-based Clustering: Foundations
• Density-based Clustering: Foundations /2
• Density-based Clustering: Foundations /3
• Density-reachability and Density-connectivity
• Density-reachability

ML U14: DBSCAN Clustering

• DBSCAN
• Clustering Approach
• Abstract DBSCAN Algorithm
• DBSCAN Algorithm
• DBSCAN Algorithm /2
• DBSCAN Algorithm /3
• DBSCAN in Context

ML U15: Parameterization of DBSCAN

• DBSCAN Parameterization
• Choosing DBSCAN parameters
• Choosing DBSCAN parameters /2
• Choosing DBSCAN parameters /3

ML U16: Extensions and Variations of DBSCAN Clustering

• DBSCAN Extensions
• Generalized Density-based Clustering
• Grid-based Accelerated DBSCAN
• Anytime Density-Based Clustering (AnyDBC)
• Hierarchical DBSCAN* (HDBSCAN*)
• Improved DBSCAN Variations

ML U17: OPTICS Clustering

• OPTICS Clustering
• Density-based Hierarchical Clustering
• Density-based Hierarchical Clustering /2
• OPTICS Clustering
• Cluster Order
• OPTICS Algorithm

ML U18: Cluster Extraction from OPTICS Plots

• Cluster Extraction from OPTICS Plots
• OPTICS Reachability Plots
• Extracting Clusters from OPTICS Reachability Plots
• Role of the Parameters and minPts

ML U19: Understanding the OPTICS Cluster Order

• Understanding the OPTICS Cluster Order
• Properties of the OPTICS Cluster Order
• Cluster Order as Serialized Spanning Tree
• OPTICS as Density Spanning Trees
• Cluster Order to Dendrograms

ML U20: Spectral Clustering

• Spectral Clustering
• Minimum Cuts
• Graph Laplacian
• From Clustering Graphs to Clustering Data
• Spectral Clustering
• Spectral Clustering is Related to DBSCAN

ML U21: Biclustering and Subspace Clustering

• Biclustering and Subspace Clustering
• Biclustering & Subspace Clustering
• Bicluster Patterns
• Density-based Subspace Clustering
• Subspace Clustering with Apriori-Style Search
• Correlation Clustering
• 4C: Computing Correlation Connected Clusters
• Hough Transform
• CASH: Robust Clustering in Arbitrarily Oriented Subspaces

ML U22: Further Clustering Approaches

• Further Clustering Approaches
• CURE Clustering
• ROCK Clustering
• CHAMELEON
• Affinity Propagation Clustering
• Other Density-based Clustering Algorithms
• Further Clustering Approaches